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PREFACE 


This  report  describes  the  development  of  instructional  strategies  for  training  complex 
decision-making  skills  to  Logistics  Command  and  Control  (LC^)  personnel.  The  work  was 
performed  in  support  of  the  “Desktop  Training  for  Logistics  Command  and  Control  (LC^” 
research  and  development  effort.  This  project  is  being  accomplished  under  Contract 
No.  F33615-91-C-0007,  with  Systems  Engineering  Associates  (SEA),  San  Diego,  CA. 
Management  of  this  effort  is  provided  by  the  Armstrong  Laboratory,  Human  Resources 
Directorate,  Technical  Training  Research  Division,  Instructional  Systems  Branch  (AL/HRTD). 
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SUMMARY 


Logistics  Command  and  Control  (LC^)  units  must  ensure  that  core  and  augmentee 
personnel  are  fully  trained  in  the  critical  combat  skill  of  decision  making.  However,  existing 
training  capabilities  are  viewed  as  inadequate.  They  consist  primarily  of  expensive  and 
manpower-intensive  exercises  which  afford  only  sporadic  training  opportunities;  consequently 
they  are  insufficient  in  ensuring  personnel  achieve  and  maintain  the  skill  levels  required  for 
successful  combat  operations.  Thus,  the  need  for  more  accessible,  more  affordable,  and  less 
manpower-intensive  training  continues  to  exist. 

In  May  1989,  the  Air  Force  Logistics  Plans  and  Programs  Directorate  (HQ  USAF/LGX) 
requested  that  the  Air  Force  Human  Resources  Directorate  (HR)  develop  an  improved  training 
technology  for  Logistics  Command  and  Control  Centers  throughout  the  United  States  Air  Force 
(USAF).  The  objective  was  to  provide  a  means  of  training  logistics  decision  makers  to  work 
with  critical  information  and  achieve  the  best  use  of  resources^  This  tasking  originated  under  a 
Memorandum  of  Agreement  between  HQ  USAF/LGX,  Air  Force  Systems  Command  (now  the 
Air  Force  Materiel  Command),  and  HR. 

In  response,  HR  let  a  contract  to  Systems  Engineering  Associates  (SEA)  to  produce  a 
desktop  decision  trainer  which  will  provide  individual  instruction  and  enable  students  to  practice 
solving  realistic  logistics  problems  in  a  Logistics  Readiness  Center  environment.  The  project, 
which  began  in  February  1992,  will  be  completed  in  February  1997. 

The  objectives  of  the  effort  are  to:  (1)  identify  and  develop  instructional  strategies  for 
training  decision-making  skills;  (2)  identify  and  develop  decision-making  models  to  serve  as  a 
basis  for  training;  and  (3)  develop  an  experimental  computer-based  training  prototype  that 
combines  decision-making  lessons  with  simulation  environments  to  enable  logistics  personnel  to 
experience  the  same  kinds  of  problems  and  situations  encountered  in  the  operational 
environment.  The  prototype  will  enable  them  to  practice  their  decision-making  skills  and  obtain 
feedback  on  their  performance. 
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I.  INTRODUCTION 


This  report  presents  the  development  of  an  instructional  strategy  for  training  decision¬ 
making  skills.  It  first  describes  the  training  problem  to  be  solved,  then  presents  the  theoretical 
foundations  for  the  solution.  Next,  the  theoretical  foundations  are  applied  to  the  problem  and  a 
detailed  training  methodology  for  problem  solving  is  developed.  The  training  problem  is 
described  as  a  need  to  provide  training  for  decision  makers  in  Logistics  Command  and  Control 
(LC^),  a  need  that  is  not  adequately  being  met  due  to  the  lack  of  an  affordable  and  effective 
methodology. 

The  theoretical  foundations  for  solving  the  training  problem  were  derived  by  addressing 
the  following  questions: 

•  What  is  decision  making? 

•  How  do  people  perform  the  decision-making  task? 

•  How  do  people  learn  to  perform  the  decision-making  task? 

•  What  can  instructional  design  theories  or  models  tell  us  about  teaching  the  decision¬ 
making  task? 

The  answers  to  these  questions  were  applied  to  the  specific  requirement  of  training  LC^  decision 
makers.  The  result  was  a  specification  for  a  training  methodology  that  will  be  implemented 
during  the  follow-on  phases  of  the  project. 

The  problem  solution  was  developed  in  an  eclectic  fashion.  No  attempt  was  made  to 
adopt  and  instantiate  one  particular  theoretical  framework.  Rather,  the  intent  was  to  bring  the 
best  thinking  and  empirical  evidence  to  bear  on  the  problem  and  to  synthesize  this  knowledge 
into  an  internally  logical  and  consistent  conceptual  framework.  This  framework  would  serve  as 
an  empirically  verifiable  foundation  for  the  design  and  development  of  a  training  system. 
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11.  PROBLEM 


LC^  and  Tactical  Command  and  Control  (TC^)  units  have  a  common  training 
requirement:  They  both  must  ensure  that  core  and  augmentee  personnel  are  folly  trained  in  the 
critical  combat  skills  of  decision  making  in  their  respective  domains.  The  commonality  of 
training  requirements  has  been  documented  by  Schwaninger,  Malin,  and  Gvunienny  (1991),  who 
surveyed  the  training  requirements  for  LC^  personnel  using  methodologies  pioneered  by  Brecke, 
Jacobs,  and  Krebs  (1988).  Schwaninger  et  al.  (1991)  foirnd  that  the  types  of  personnel  and  skills 
that  need  to  be  trained  in  LC^  are  the  same  as  those  identified  for  TC^  in  the  earlier  Brecke  et  al. 
(1988)  study.  Also  common  to  both  C^  communities  is  the  fact  that  the  existing  training 
capabilities  are  inadequate.  They  currently  consist  primarily  of  expensive  and  manpower¬ 
intensive  exercises  which  afford  only  sporadic  training  opportunities  that  are  insufficient  to 
achieve  and  maintain  the  personnel  skill  levels  required  for  successful  combat  operations.  The 
need  for  more  accessible,  more  affordable,  and  less  manpower-intensive  training,  documented  by 
both  aforementioned  studies,  continues  to  exist. 

The  common  shortfall  in  training  capability  indicates  that  both  LC^  and  TC^  units  might 
benefit  from  the  same  training  solution:  a  training  technology  based  on  desktop  computers.  This 
technology,  which  began  to  emerge  during  the  early  1980’s,  uses  the  increasing  power  of 
desktop-based  computer  hardware  and  software  technology  to  produce  credible  simulation 
environments  within  which  the  battle  manager  trainee  can  experience  the  same  kinds  of  problem 
and  decision-making  situations  encountered  in  the  operational  environment. 

The  feasibility  of  generating  realistic  tactical  simulation  environments  for  training  on 
desktop-class  hardware  has  been  demonstrated  with  the  SuperKEATS  system  developed  by  the 
Air  Force  Hxraian  Resources  Laboratory  (AFHRL)  during  an  earlier  effort  (Brecke  &  Young, 
1990).  The  SuperKEATS  system  achieves  a  realistic  TC^  simulation  by  means  that  are  fairly 
simple  compared  with  the  complex  means  employed  in  large-scale  simulation  environments  that 
are  widely  used  to  support  exercises.  While  this  is  an  encouraging  result  and  a  necessary  first 
step  in  solving  the  training  problem,  it  is  not  sufficient  because  it  addresses  only  one  side  of  the 
training  methodology  issue:  the  delivery  strategy. 

According  to  Reigeluth  (1983),  a  complete  specification  of  an  instructional  method 
requires  the  specification  of  three  types  of  strategies:  organizational,  delivery,  and  management. 
An  organizational  strategy  is  a  prescription  for  the  organization  of  instructional  macro  and  micro 
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elements.  A  delivery  strategy  defines  the  medium  used  to  convey  the  instruction.  In  our  case, 
the  delivery  strategy  is  given  or  prescribed  and  consists  of  computer-based  self-instruction 
running  on  a  desktop  personal  computer  (PC)  platform.  A  management  strategy  defines  which 
organizational  or  delivery  strategy  components  will  be  used  imder  different  conditions  during 
instruction.  Organizational  and  management  strategies  are  not  prescribed  in  our  case;  rather, 
they  must  both  be  developed  and  defined. 

Organizational  and  management  strategies  (hereafter  referred  to  as  instructional 
strategies)  cannot  (yet)  be  taken  off-the-shelf,  even  though  an  instructional  strategy  for  decision¬ 
making  skills  was  identified  by  Aagard  and  Braby  (1976).  This  strategy  has  a  great  deal  of 
appeal  and  face  validity,  but  its  empirical  and  analytical  or  theoretical  support  is  rmclear.  The 
objective  of  this  paper  and  of  this  project  is  to  take  a  fresh  look  at  the  problem  of  training 
decision-making  skills  and  to  develop  instructional  strategies  for  this  class  of  skills  on  the  basis 
of  current  empirical  and  theoretical  knowledge  from  the  following  four  sources: 

•  A  precise  and  complete  definition  of  the  type  of  task  to  be  trained. 

•  A  clear  picture  of  the  process  of  task  performance. 

•  Explanations  of  the  process  of  learning  or  skill  acquisition  for  this  type  of  task. 

•  Instructional  design  theories. 

There  are  limits  to  what  can  be  achieved  from  a  "theoretical  armchair."  Even  if  these 
combined  sources  provide  strong  guidance  towards  a  particular  instructional  strategy,  it  is 
impossible  to  guarantee  that  such  a  strategy  is  in  any  sense  optimal.  A  careful  derivation  of 
instructional  strategy  from  the  sources  listed  above  will  certainly  reduce  the  search  space,  but  the 
optimal  strategy  for  a  given  set  of  training  conditions  can  ultimately  be  found  only  by  empirical 
means.  The  training  system  produced  by  this  project  must  also  be  capable  of  serving  as  a 
research  testbed;  therefore  it  must  permit  easy  modification  of  the  instructional  strategies  by 
which  the  system  delivers  training. 

This  report  is  devoted  to  narrowing  the  search  space  for  the  ideal  or  optimal  instructional 
strategy  for  training  decision-making  skills.  The  goal  is  to  define  a  recommended  "strawman" 
strategy  and  to  identify  the  elements  of  this  strategy  that  require  empirical  validation  and/or 
exploration.  The  report  thus  provides  the  theoretical  underpinning  for  the  design  of  a  training 
system  which,  for  research  purposes,  can  be  modified  along  a  number  of  pre-specified 
dimensions. 
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III.  THEORETICAL  FOUNDATIONS 


3.1.  What  IS  Decision  Making? 

Definitions  of  decision  making  can  be  found  in  the  literature  (Anderson,  Deane, 
Hammond,  &  McClelland,  1981;  Janis  &  Mann,  1977;  Nickerson  &  Feehrer,  1975;  Webster 
1981),  but  these  definitions  generally  lack  precision  and  completeness.  To  achieve  a  precise  and 
complete  definition  for  the  class  of  decision-making  tasks,  a  review  of  common  definitions  of 
decision  making  was  performed.  The  review  resulted  in  a  three-component  description  of  the 
decision-making  process:  (1)  imcertainty  model,  (2)  time  line  model,  and  (3)  verbal  definition. 
Together  these  components  constitute  a  definition  of  decision  making  that  is  considered 
complete  and  precise  enough  for  the  purpose  of  developing  a  training  methodology. 

3.1.1.  Uncertainty  Model 

The  analysis  of  decision  making  began  with  an  assessment  of  some  common  definitions 
foimd  in  the  literature.  Anderson  et  al.  (1981)  define  decision  making  as  "selecting  and 
committing  oneself  to  a  course  of  action."  This  definition  is  clearly  in  consonance  with  the 
colloquial,  popular  imderstanding  of  the  term,  but  it  is  also  incomplete.  Missing  from  this 
definition  is  an  essential  component  that  comes  through  in  Webster's  (1981)  definition  of  the 
term:  “1.  to  arrive  at  a  solution  that  ends  uncertainty."  Taken  together,  the  Anderson  et  al.  and 
Webster  definitions  identify  decision  making  as  an  activity  that  ends  uncertainty  through  a 
commitment  to  a  particular  course  of  action.  The  notion  of  eliminating  or  reducing  uncertainty 
and  commitment  appear  to  be  essential  aspects  of  decision  making.  Working  firom  this 
foundation,  focus  was  then  placed  on  a  more  detailed  analysis  of  uncertainty  reduction.  A 
primary  goal  was  to  relate  the  cognitive  activity  of  uncertainty  reduction  to  the  affective  activity 
of  making  a  commitment. 

Decision  making  begins  with  a  perception  that  a  situation  has  developed  in  which  the 
following  pertain: 

•  Significant  action  alternatives  exist. 

•  There  is  pressure  to  commit  to  a  course  of  action. 

•  There  is  an  uncomfortable  amormt  of  uncertainty  with  respect  to  making  that 
commitment. 
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All  three  of  these  conditions  are  required  to  trigger  the  decision-making  activity.  The 
action  alternatives  need  not  be  well  articulated  or  explicit,  but  there  must  be  at  least  the  implicit 
notion  that  more  than  one  course  of  action  is  possible.  In  addition,  there  must  be  sufficient 
motivation  or  need  to  overcome  a  threshold  of  inertia  and  some  degree  of  xmcertainty. 

This  uncertainty  is  initially  a  basic  dubiety  about  what  to  do,  which  course  of  action  to 
choose,  which  alternative  to  commit  to.  This  initial  uncertainty  is  referred  to  as  primary 
uncertainty.  Primary  uncertainty  is  reduced  to  zero  as  soon  as  the  decision  is  made.  In  that 
sense,  the  Webster  definition  is  exact:  a  decision  is  a  solution  that  "ends"  uncertainty.  A 
decision  determines  which  course  of  action  is  to  be  taken,  or,  if  only  one  course  of  action  is 
imder  consideration,  it  indicates  that  a  commitment  to  "the"  course  of  action  has  been  made. 

This  resolution  relieves  stress  but  does  not  reduce  all  uncertainty  in  the  decision  problem  to  zero 
(i.e.,  Webster's  definition  is  only  partially  correct). 

Three  basic  types  of  information  are  required  for  a  decision.  The  absence  of  any  one  of 
the  three  results  in  an  imcertainty  that  differs  fi:om  primary  uncertainty.  This  type  of  uncertainty 
is  called  secondary  uncertainty.  The  decision  maker  attempts  to  reduce  secondary  uncertainty  to 
zero  prior  to  committing  to  a  particular  course  of  action.  Three  forms  of  secondary  uncertainty 
have  been  identified  in  the  literature  (Coombs,  Dawes,  &  Tversky,  1970;  Einhom  &  Hogarth, 
1985;  Kirschenbaum,  1986),  but  they  have  not  been  used  together  as  components  of  a  systematic 
definition  of  the  uncertainty  variable  in  decision  making.  The  three  types  of  secondary 
uncertainty  are: 

1 .  Situation  Uncertainty  (Ug) . 

2.  Goal  Uncertainty  (Uq). 

3.  Option  Uncertainty  (Uq). 

Situation  uncertainty  is  uncertainty  with  regard  to  the  situation  that  elicits  the  decision¬ 
making  activity.  Situation  xmcertainty  is  represented  by  questions  such  as,  "What  is  going  on?" 
or  "What  is  the  problem?"  Decision  makers'  efforts  to  deal  with  this  type  of  imcertainty  have 
been  referred  to  as  "problem  structuring"  or  "hypothesis  formation"  (Nickerson  &  Feehrer, 

1975). 


Goal  uncertainty  is  uncertainty  as  to  the  goals  or  objectives  that  are  to  be  achieved  by  the 
decision.  Goal  uncertainty  is  represented  by  questions  such  as,  "What  do  I  want  to  achieve?"  or 
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"Why  are  we  in  this  business?"  or  "What  do  I  want  to  get  out  of  this?"  Goals  or  objectives  may 
compete  with  one  another,  thus  forcing  the  decision  maker  to  prioritize  them.  This  prioritization 
often  requires  an  introspective  or  communal  clarification  process  (or  both). 

Option  uncertainty  is  uncertainty  as  to  the  options  or  action  alternatives  available  to 
achieve  the  desired  goals  in  the  given  situation.  Option  uncertainty  is  more  complex  than 
situation  and  goal  uncertainty.  Three  distinct  subtypes  of  option  uncertainty  can  be  identified; 

1.  Set  Completeness  Uncertainty  (Uqs):  Uncertainty  as  to  whether  the  set  of  options 
under  consideration  is  complete.  This  type  of  uncertainty  is  represented  by 
questions  such  as,  "Isn't  there  another  way?"  or  "What  else  can  we  do?"  or  "Have  I 
really  considered  all  the  possible  alternatives?" 

2.  Effects  Uncertainty  (Uoe)-  Uncertainty  about  the  effects  or  results  of 
implementing  a  given  option.  When  multiple  options  are  available  and/or 
considered,  the  decision  maker  may  be  very  certain  about  the  possible  effects  of 
some  and  completely  uncertain  about  the  effects  of  others.  By  the  same  token,  there 
may  be  certainty  about  some  effects  of  a  particular  option  and  much  less  certainty 
about  other  effects  of  the  same  option.  Effects  rmcertainty  is  represented  by 
questions  such  as,  "What  is  this  going  to  do  for  me?"  or  "Will  the  cure  be  worse 
than  the  disease?"  or  "Will  this  action  have  the  desired  results?" 

3.  Feasibility  Uncertainty  (Uqf):  Uncertainty  with  respect  to  whether  a  given  option 
can  be  implemented  as  desired.  Feasibility  uncertainty  is  represented  by  questions 
such  as,  "How  old  is  this  bungee  cord?"  or  "Can  I  say  it  with  a  straight  face?"  or 
"What  if  the  guard  screams  before  I  can  cut  his  throat?" 

Both  primary  and  secondary  uncertainty  are  usually  present  to  some  degree  in  any 
decision  problem.  The  overall  uncertainty  in  a  decision  problem  is  thus  an  aggregate  of  all  the 
types  of  uncertainty  that  can  be  represented  with  the  tree-like  structure  shown  in  Figure  1.  This 
figure  and  its  underlying  notions  are  known  as  the  "vmcertainty  model." 

Prior  to  committing  to  a  particular  course  of  action,  the  decision  maker  attempts  to  reduce 
all  these  types  of  secondary  uncertainty.  The  level  to  which  the  uncertainty  must  be  reduced  for 
the  commitment  to  occur  is  basically  a  function  of  what  is  at  stake  for  the  decision  maker,  the 
available  time  and  the  costs  of  obtaining  further  information. 
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Figure  1 .  Uncertainty  Model 


3.1.2.  Time  Line  Model 


Decision  making  is  also  a  cognitive  process  that  can  be  described  in  terms  of  a  time  line. 
This,  strangely  enough,  has  not  been  done  before.  The  decision-making  process  has  been 
decomposed  into  phases  and/or  sequences  of  tasks  (Nickerson  &  Feehrer,  1975;  Schrenk,  1969). 
But  the  main  focus  of  these  decompositions  is  usually  on  the  nature  of  each  phase  rather  than  on 
temporal  relationships,  and  the  decompositions  invariably  carry  a  strong  prescriptive  (and 
therefore  somewhat  arbitrary)  flavor  rather  than  a  strictly  descriptive  one. 

The  time  line  model  pictured  in  Figure  2  reveals  the  temporal  anatomy  of  the  decision 
task.  The  time  increments  pictured  could  be  anything  from  seconds  to  years.  The  diagram 
shows  a  "normal"  (or  perhaps  "ideal")  decision-making  task.  Many  variations  in  the 
relationships  between  the  phases  and  milestones  are  possible,  each  generating  particular  demands 
on  the  decision  maker. 
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Figure  2.  Time  Line  Model  for  Decision-Making  Tasks 


TIME 


Most  of  the  diagram  is  probably  self-explanatory,  however  the  following  milestone  labels 
require  explanation: 


Objective  Choice  Point  (OCP):  The  OCP  is  the  point  at  which  the  course  of  events 
surrounding  the  decision  maker  has  produced  a  situation  in  which  both  the  opportunity  and  the 
need  for  a  decision  are  present.  Prior  to  this  point,  a  decision  can  be  anticipated  and  can  even  be 
made  in  anticipation  (as  in  a  plan),  but  it  need  not  be  made  and  it  cannot  be  implemented. 


Recognition  Point  (RP):  The  RP  is  the  point  at  which  the  prospective  decision  maker 
recognizes  a  need  and  an  opportunity  for  a  decision. 

Decision  Point  (DP):  The  DP  is  the  point  at  which  the  decision  is  made,  the  "moment  of 
truth."  More  precisely,  at  this  point  the  process  of  decision  making  proper  has  come  to  an  end 
and  the  decision  maker  has  committed  to  a  particular  option. 


Default  Point  (DefP):  The  DefP  is  the  point  at  which  the  need  or  opportunity  (or  both) 
for  the  decision  disappears  because  it  literally  has  been  "overcome  by  events."  This  is  where 
whatever  will  happen  without  interference  by  the  decision  maker  will  actually  start  to  happen 
(i.e.,  the  default  course  of  events  will  occur).  This  may,  of  course,  trigger  another  OCP,  but  that 
OCP  will  be  the  start  of  another,  different  decision  evolution. 

Figure  2  shows  that  the  decision-making  process  begins  with  the  OCP,  which  occurs 
when  both  the  opportunity  and  the  need  for  a  decision  appear  in  a  given  domain.  The  OCP  may 
be  preceded  by  an  anticipatory  planning  or  incubation  period.  The  RP  marks  the  point  at  which 
the  prospective  decision  maker  recognizes  the  opportunity  and  need  for  a  decision  and  the  start  of 
the  cognitive  activities  that  are  the  heart  of  the  decision-making  process.  These  activities  may 
either  start  immediately  after  recognition  or  after  some  delay,  and  they  occur  during  one  or  more 
working  episodes  interrupted  by  work  suspense  periods.  The  last  working  episode  ends  with  the 
DP.  The  DP  should  be  reached  before  the  DefP;  implementation  of  the  decision  should  also  at 
least  start  before  the  DefP.  There  is  a  possible  delay  between  the  DP  and  the  start  of 
implementation.  Feedback  begins  following  the  implementation  (i.e.,  there  is  another  possible 
delay)  and  can  continue  indefinitely  after  the  implementation  process  has  ceased.  The  end  of  the 
feedback  process  marks  the  end  of  the  entire  decision-making  episode. 

3.1.3.  Definition  and  Discussion 

The  uncertainty  model  and  the  time  line  model  can  be  combined  into  a  verbal  definition 
of  decision  making. 


Decision  Making  Defined 

1 .  Decision  making  is  a  cognitive  and  elective  process  that  is  triggered  by  the 
perception  of  the  following: 

•  The  possibility  of  more  than  one  course  of  action. 

•  A  certain  amount  of  pressure  to  "do  something." 

•  A  more  or  less  complex  aggregate  of  primary  and  secondary  imcertainty. 

2.  The  process  begins  with  the  OCP,  ends  with  the  feedback  finish  point,  and 
incorporates  four  subprocesses: 

•  Recognition. 

•  Uncertainty  Reduction. 


9 


•  Implementation. 

•  Feedback. 

3.  Uncertainty  reduction,  bracketed  by  the  RP  and  the  DP,  consists  of  activities  designed 
to  reduce  all  components  of  secondary  uncertainty  to  levels  where  commitment  to  a 
particular  course  of  action  becomes  possible. 

4.  Commitment  to  a  particular  course  of  action  constitutes  the  decision.  The  decision 
reduces  primary  uncertainty  to  zero,  while  secondary  uncertainty  remains  at  the  level 
it  had  when  the  commitment  occurred  (residual  uncertainty). 

5.  Residual  uncertainty  can  only  be  reduced  further  by  feedback  from  the  enviromnent. 
Such  feedback  can  only  occur  after  the  decision  has  been  implemented,  and  it  may  be 
incomplete,  difficult  to  interpret,  delayed,  and  difficult  to  attribute  to  a  particular 
decision.  Feedback  may  even  increase  residual  imcertainty  as  more  information  is 
obtained. 

The  act  of  choice  or  commitment  in  decision  making  always  occurs  in  the  face  of  residual 
secondary  uncertainty.  If  it  did  not,  there  would  have  to  be  complete  certainty  of  what  the 
situation  is,  what  the  goals  are,  and  which  option  optimizes  the  achievement  of  these  goals;  such 
certainty  would  not  require  a  decision.  In  such  a  case,  the  problem  is  solved  and  the  activities 
that  lead  to  the  solution  can  be  called  problem  solving  but  not  decision  making.  Since  some 
secondary  uncertainty  remains,  the  act  of  choice  and  commitment  requires  more  than 
ratiocination;  it  requires  a  push  of  affective  energy  to  accomplish  a  daring  leap  into  the  unknown 
and  presently  unknowable. 

This  interpretation  of  the  decision-making  phenomenon  essentially  eliminates  the 
distinction  between  risky  and  riskless  decision  making  and  between  decision  making  under 
uncertainty  and  decision  making  imder  certainty.  It  asserts  that  decision  making  is  always  risky, 
always  a  gamble,  and  always  occurs  under  uncertainty.  Without  risk  and  uncertainty,  the  process 
is  simply  one  of  "taking  an  action"  rather  than  making  a  decision. 

In  addition  to  clarifying  the  distinction  between  choice  and  decision,  this  interpretation 
also  clarifies  the  distinction  between  judgment  and  decision  making.  According  to  Fischoff 
(1989),  judgment  is  needed  to  extract  information  from  an  uncertain  environment,  while  decision 
making  is  needed  to  extract  a  course  of  action  from  those  judgments  in  order  to  achieve  some 
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goal(s).  The  information  search  that  is  undertaken  to  reduce  secondary  uncertainty  must  be 
supplemented  by  judgment  to  fill  in  the  gaps  between  the  morsels  of  "hard"  information  that  are 
accessible  to  the  decision  maker. 

In  this  context,  activities  which  produce  a  diagnosis,  a  situation  assessment,  or  a 
prediction  as  their  end  result  are  often  wrongly  referred  to  as  decision  making.  The  aspect  of 
uncertainty  reduction  is  clearly  present  in  such  activities,  but  the  other  essential  component  of 
decision  making,  (i.e.,  the  commitment  to  a  course  of  action)  is  missing.  For  example,  a  doctor 
does  not  decide  a  diagnosis  of  cancer,  rather  a  judgment  is  made  based  on  symptoms.  The  doctor 
makes  a  decision  only  if  this  judgment  is,  explicitly  or  implicitly,  tied  to  a  commitment  to  a 
particular  course  of  therapeutic  intervention.  An  intelligence  analyst  who  predicts  a  particular 
enemy  route  of  approach  does  not  decide  on  that  approach  --  the  enemy  does.  A  judgment  is 
made  based  on  a  number  of  possible  approaches  that  appear  most  likely.  If  the  analyst  is 
concerned  about  their  personal  credibility  and  standing  with  their  comm^der,  they  may  abandon 
their  best  objective  judgment  and  commit  to  a  particular  prediction  that  they  believe  the 
commander  "wants  to  hear."  The  analyst  has  indeed  made  a  decision,  but  it  is  one  regarding  their 
own  welfare,  not  one  relating  to  possible  enemy  routes  of  approach. 

Finally,  it  is  clear  that  decision  making  is  an  activity  with  very  strong  affective 
components.  The  decision-making  process  is  not  initiated  unless  there  is  some  motivational 
pressure,  uncertainty  reduction  requires  judgment  that  is  subject  to  distortions  caused  by  stress 
and  other  affective  factors,  and,  finally,  the  process  is  not  terminated  unless  there  is  sufficient 
affective  impetus  to  overcome  emotional  barriers  to  commitment  in  the  face  of  remaining 
uncertainty. 

3.1.4.  Models  Applied  to  Examples 

To  illustrate  the  uncertainty  and  time  line  models,  we  apply  them  to  three  examples: 
decisions  made  by  dispatchers  in  taxi  companies,  a  decision  a  supply  logistician  may  have  to 
make  during  a  conflict,  and  a  car-buying  decision  made  by  a  typical  consumer.  The  examples 
are  shown  (in  a  side-by-side  arrangement  to  permit  convenient  comparisons  in  Appendix  A). 
Figure  3  presents  a  graphic  image  summarizing  some  points  from  Appendix  A. 
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Figure  3.  Graphic  "Uncertainty  Profiles"  for  Three  Types  of  Decision-Making  Tasks 


Uncertainty 


3.1.5.  Taxonomic  Issues 

The  models  and  the  definition  supplied  above  suggest  taxonomic  possibilities  that  are 
both  practically  useful  and  theoretically  significant.  For  example,  one  might  examine  a  type  of 
decision  task  for  which  options  must  be  developed  and  contrast  it  with  another  type  of  decision 
task  for  which  the  options  are  clear  but  the  goals  are  not.  The  task  type  requiring  option 
generation  relies  on  creativity  during  task  performance.  Training  such  a  task  would  require  a 
training  system  capable  of  evaluating  novel  approaches  to  a  class  of  decision  problems,  a 
capability  not  easily  developed  in  a  computer-based  system.  In  the  second  task  type,  the  decision 
maker  has  a  clear  set  of  options  (i.e.,  certainty  regarding  the  options,  how  feasible  they  are,  and 
what  the  effects  will  be),  but  the  goals  are  not  firmly  articulated  and  prioritized.  This  task  type 
requires  no  creativity  or  originality  and  a  training  system  would  merely  have  to  be  able  to 
recognize  a  correct  choice  from  existing  options.  This  capability  is  realized  relatively  easily  in  a 
computer-based  training  system.  These  two  types  of  tasks  obviously  require  fundamentally 
different  cognitive  activities  to  reduce  secondary  uncertainty;  they  also  require  different  learning 
and  instructional  strategies  during  training. 

Not  only  do  the  qualitative  differences  between  uncertainty  types  have  taxonomic  va,lue, 
the  quantitative  aspects  of  uncertainty  also  provide  valuable  dimensions  of  task  discrimination. 
For  example,  a  decision  task  that  is  normally  easy  to  perform  for  personnel  of  a  given 


12 


proficiency  level  can  become  a  completely  different  task  (i.e.,  a  "Mission  Impossible")  when  an 
information  channel  dries  up  (e.g.,  when  a  telephone  line  gets  cut).  In  such  a  case,  the  amoimt  of 
uncertainty  suddenly  rises  substantially  above  normal  levels.  Both  Howard  (1968)  and  Brecke 
(1982)  refer  to  the  amount  of  uncertainty  as  an  important  dimension  determining  the  nature  of  a 
decision-making  task  or  of  a  judgment  task.  Both  researchers  see  uncertainty  as  one  of  three 
dimensions  that  together  determine  the  nature  of  a  decision-making  task.  The  other  two 
dimensions  are  complexity  and  time  constraint  (see  Figure  4). 


Figure  4.  Dimensions  of  the  Decision  Task 


Additional  key  variables  are  frequency  of performance  and  task  importance  (which  is 
most  likely  some  aggregate  of  personal  and  institutional  variables).  These  types  of  variables  are 
routinely  used  by  training  designers  because  they  provide  relevant  criteria  for  training  design 
decisions. 

Some  decision  tasks  are  clearly  very  important,  complex,  and  absolutely  buried  in 
imcertainties.  However,  if  they  occur  only  once  or  twice  in  a  lifetime  (such  as  deciding  on  a 
partner  in  matrimony),  task-specific  training  may  not  have  any  beneficial  effect  (such  as 
lowering  the  probability  of  divorce).  On  the  other  hand,  some  decision-making  tasks,  although 
they  occur  frequently,  are  so  simple  and  trivial  that  any  effort  to  provide  formal  training  for  them 
would  be  a  waste  of  time.  Between  these  extremes  lie  tasks  that  are  the  "bread  and  butter"  of 
decision  makers  everywhere;  tasks  that  occur  frequently,  represent  a  qualitatively  and 
quantitatively  significant  portion  of  a  given  job  performer's  work,  and  are  complex  enough  to 
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require  nontrivial  learning  time  and  effort.  Clearly,  training  developers  and  management  should 
give  such  tasks  priority.  Furthermore,  this  line  of  reasoning  facilitates  intelligent  allocation  of 
training  resources. 

The  uncertainty  and  time  line  models  allow  for  development  of  descriptive  and 
distinctive  decision  task  profiles  for  real-world  decision-making  tasks  (e.g.  see  Figure  3).  These 
profiles  can  then  be  emulated  by  practice  decision-making  tasks  presented  during  training. 
Therefore,  the  models  are  not  only  useful  as  theoretical  taxonomic  tools  but  also  as  instruments 
for  ensuring  that  training  tasks  are  structured  like  real-world  tasks.  The  models  can,  in  other 
words,  make  essential  contributions  to  positive  transfer  of  training.  They  can  be  used  along  with 
other  factors  to  select  tasks  for  training  and,  once  selection  has  occurred,  to  ensure  training 
effectiveness. 

3.1.6.  Training  Considerations 

The  analysis  of  the  decision-making  process  produced  an  arguably  better  imderstanding 
of  the  decision-making  task.  The  question  now  is  whether  it  is  possible  to  draw  some 
conclusions  that  contribute  to  an  optimal  training  methodology  for  the  task.  Does  this  set  of 
models  and  the  definition  help  determine  what  the  content  of  decision  training  should  be  or  what 
the  instructional  strategy  should  or  should  not  be? 

If  the  models  (and  the  definition)  are  enlightening  and  contribute  to  a  better 
understanding  of  the  task,  it  might  be  argued  that  students  will  benefit  fi-om  knowing  more  about 
the  general  task  (i.e.,  from  instruction  in  this  type  of  meta-content).  They  may  further  benefit 
fi-om  being  able  to  recognize  the  specific  forms  of  the  time  line  model  and  the  uncertainty  model 
in  the  types  of  decision-making  tasks  that  occur  in  the  technical  domain  within  which  they  will 
be  called  upon  to  make  decisions. 


Instructional  Guideline  No.  1 

I&  Include  in  the  training  system  the  capability  to  train  students  in  the  use  of  the  uncertainty 
and  time  line  models. 


A  more  complete,  precise,  and  detailed  definition  of  the  type  of  decision  making  provides 
clues  for  training  content  —  that  is,  WHAT  should  be  taught.  It  also  provides  some  important 
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clues  regarding  training  strategy  (i.e.,  regarding  HOW  this  content  and  the  associated  skills 
should  be  taught). 

Instructional  Guideline  No.  2 

Ensure  that  practice  problems  have  the  same  uncertainty  and  time  line  profiles  as  the 
target  job  tasks. 

Guideline  3  is  a  corollary  to  guideline  2: 

Instructional  Guideline  No.  3 

I&  Ensure  that  training  provides  for  a  practice  environment  that  features  the  s^e  types  of 
information  sources  and  requires  the  same  kinds  of  information  access  procedures  as  the 
target  environment. 

Finally: 

Instructional  Guideline  No.  4 

I&  During  practice,  gradiially  increase  time  constraint,  complexity,  and  uncertainty  to  levels 
encovmtered  in  the  target  j  ob  environment. 

3.2.  How  Do  People  Perform  the  Decision-Making  Task? 

3.2.1.  Introduction 

The  answer  to  the  question  posed  by  the  title  of  this  section  has  been,  and  continues  to  be, 
the  cardinal  goal  of  descriptive  research  in  decision-making.  The  ideal  answer  is  a 
comprehensive  descriptive  model  that  accounts  for  the  entire  process  of  decision-making  ~  from 
the  emergence  of  a  decision  problem  (i.e.,  from  the  OCP)  to  the  conclusion  of  the  feedback 
process  (i.e.,  the  FFP).  The  ideal  model  also  has  to  accommodate  the  view  that  the  decision¬ 
making  process  is  not  a  unitary  phenomenon,  rather  it  varies  with  the  characteristics  of  the 
decision-making  task  as  well  as  with  the  characteristics  of  the  decision  maker. 
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During  the  more  than  40  years  of  rather  vigorous  research  in  decision  making,  such  a 
model  has  been  slow  to  emerge.  Much  of  that  time  was  dominated  by  a  prescriptive  rather  than 
descriptive  view  of  the  decision-making  process.  This  view  was  founded  on  the  axiomatization 
of  decision  theory  by  von  Neumann  and  Morgenstem  (1947)  and  was  manifested  in  a  large 
number  of  studies.  These  studies  compared  human  decision-making  performance  on  contrived 
decision  tasks  in  laboratory  settings  with  logical  or  rational  ideals  of  decision  making  expressed 
by  mathematical  models  such  as  Bayes'  theorem  or  Expected  Utility  theory.  This  research, 
although  widely  criticized  for  its  dubitable  generalizability  to  real-world  decision  making  (for 
example,  see  Fischhoff,  1989),  produced  many  valuable  insights  into  the  weaknesses  and  foibles 
of  humans  as  decision  makers.  However,  the  preoccupation  of  this  research  paradigm  with 
human  suboptimality  likely  prevented  an  unbiased  examination  of  actual,  real-world  decision 
making  and  the  development  of  a  complete  descriptive  model  as  defined  above. 

In  the  last  five  years,  the  pendulum  has  begim  to  swing  more  towards  purely  descriptive 
research.  As  a  result,  evidence  is  accumulating  which  indicates  that  in  decision  making,  as  in 
other  cognitive  tasks,  experts  go  through  a  process  that  is  qualitatively  and  quantitatively  very 
different  from  the  process  novices  go  through.  The  process  followed  by  experts  appears  to  have 
little  resemblance  to  the  analytical  models  inspired  by  decision  theory,  whereas  novices  are 
forced  to  use  such  "weak"  general  logical  methods.  This  more  recent  research  has  brought  the 
heretofore  rather  elusive  goal  of  a  valid  descriptive  model  much  closer  to  reality. 

The  following  paragraphs  present  the  contributions  of  the  earlier,  prescriptively  oriented 
research  to  an  account  of  human  performance  in  decision-making  tasks.  As  more  recent  and 
purely  descriptive  research  is  discussed,  the  consistencies  between  these  research  findings  with 
cognitive  task  models  proposed  by  Clancey  (1987)  and  Rasmussen  (1988)  are  pointed  out,  and  a 
process  model  based  on  the  uncertainty  and  time  line  models  presented  in  the  preceding  section 
is  suggested.  Finally,  instructional  design  guidelines  based  on  an  understanding  of  the  task 
performance  process  in  decision  making  is  provided. 

The  following  discussion  is  limited  to  unaided  decision  making.  This  seems  to  be  a 
particularly  necessary  clarification  in  the  context  of  military  decision  making  that  has  been  the 
focus  of  vigorous  research  and  development  in  decision  aiding.  Sheridan  and  Verplanck  (1978) 
have  distinguished  10  levels  of  allocation  of  decision  tasks  between  humans  and  computers.  At 
Level  1,  the  "human  does  the  whole  job  up  to  the  point  of  turning  it  over  to  the  computer  to 
implement."  At  Level  10,  the  "computer  does  the  whole  job  if  it  decides  it  should  be  done,  and  if 
so,  tells  the  human  if  it  decides  the  human  should  be  told."  Of  interest  in  this  project  is  the 
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description  of  human  decision-making  performance  at  Level  1,  where  humans  do  it  the  "old- 
fashioned  way." 

3.2.2.  Contributions  of  Earlier,  Prescriptively  Oriented  Research 

When  Nickerson  and  Feehrer  published  a  comprehensive  review  of  the  research  in 
decision  making  in  1 975,  their  intent  was  the  same  intent  as  that  of  this  research  effort:  to  derive 
prescriptive  training  principles.  The  authors  concluded  that  despite  the  voluminous  literature  on 
decision-making  research  the  number  of  studies  that  explicitly  addressed  the  question  of  exactly 
what  should  be  taught  and  how  the  teaching  can  best  be  accomplished  is  remarkably  small. 
Therefore,  they  looked  beyond  such  studies  at  a  rather  broad  cross  section  of  the  general 
decision-making  research  literature.  Nevertheless,  the  yield  of  prescriptive  training  principles 
was  meager  at  best.  The  following  paragraphs  summarize  the  results  of  decision-making 
research  up  to  1975,  relying  heavily  on  Nickerson  and  Feehrer's  landmark  report. 

Nickerson  and  Feehrer  (1975)  synthesized  their  findings  in  a  model  that  conceptualized 
the  decision-making  process  as  consisting  of  eight  subtasks,  phases,  or  aspects  and  indicated 
clearly  that  this  conceptualization  has  an  element  of  arbitrariness  about  it  —  as  does  any  other. 
Their  report  is  structured  around  these  eight  aspects  and  presents  for  each  what  was  known  about 
it  at  that  time  and  the  training  guidelines  one  might  derive  from  that  knowledge.  This  model  is 
provided  in  graphic  form  in  Figure  5.  This  figure  shows  a  central  "executive"  aspect  or  function 
that  seems  to  be  implied,  but  is  not  explicitly  identified,  by  Nickerson  and  Feehrer. 

Figure  5.  Nickerson  and  Feehrer’s  Descriptive  Model  of  Decision  Making 
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Rather  than  paraphrasing  Nickerson  and  Feehrer's  discussions  of  each  aspect  in  an 
abbreviated  form,  the  essential  conclusions  are  provided  in  Table  1.  Nickerson  and  Feehrer's 
report  demonstrates  that,  at  the  time  the  report  was  prepared,  the  decision-making  knowledge 
base  was  not  complete  enough  for  researchers  to  derive  from  it  any  sort  of  coherent  set  of 
prescriptions  for  training.  In  a  more  recent  review,  Boff  and  Lincoln  (1988)  compiled  a  table  of 
human  decision-making  characteristics  that  presents  findings  similar  to  those  of  the  Nickerson 
and  Feehrer  report.  Their  findings  are  provided  in  Table  2. 


Table  1 .  Findings  in  Nickerson  and  Feehrer  (1975) 


INFORMATION 
GATHERING 
(=  acquiring  information 
to  reduce  uncertainty) 

•  Limited  capacity  to  assimilate  information 

•  Purchases  too  much  information  when  uncertainty  is  low  and 
vice  versa 

•  Consults  unreliable  sources  too  frequently  and  vice  versa 

•  Train  to  recognize  limitations  to 
assimilate  information 

DATA  EVALUATION 
(=  judging  the  "quality" 
of  data) 

•  Evaluates  reliability  and  validity  of  data  informally,  not  in 
quantitative  terms 

•  Very  low  levels  of  inter-subject  agreement 

•  Establish  formal  mapping  between 
verbal  terms  and  numeric  terms 

•  Train  that  mapping 

PROBLEM 
STRUCTURING 
(=  make  the  structure  of 
the  problem  explicit) 

•  Typical  approach  appears  to  be  from  general  to  specific 

•  Little  known  aspect  of  decision  making 

•  Provide  model  structures  for  specific 
domains 

•  Train  people  to  solve  problems  in 
different  ways 

•  Train  the  general-to-specific  approach 

HYPOTHESIS 
GENERATION 
(=  generate  hypotheses 
on  a  situation) 

•  Generally  quite  adept  at  generating  hypotheses 

•  Intelligeiltly  and  flexibly  adapts  hypothesis  formulation  strategy 
to  situational  constraints 

•  Has  a  tendency  to  persevere  with  a  disconfiimed  hypothesis 

•  Train  to  avoid  perseverance 

•  Promote  creative  thinking 

HYPOTHESIS 
EVALUATION 
(=  Judging  probability  of 
the  truth  of  a  hypothesis) 

•  Prone  to  logical  fallacies 

.  •  Pretty  good  at  estimating  relative  frequencies,  means,  and 
medians 

•  Representativeness  heuristic:  similarity  is  an  index  of  class 
membership 

•  Availability  heuristic:  ease  of  remembering  instances  is  an 
indication  of  frequency  of  occurrence 

•  Conservatism:  extracts  less  information  from  data  than  the  data 
contain 

•  Partiality:  gives  more  credence  to  evidence  that  confirms  a 
favored  hypothesis  than  to  evidence  that  disconfirms  it 

•  Train  decision  makers  to  be  aware  of 
the  heuristics  and  to  guard  against 
their  misapplication 

•  Train  to  avoid  conservatism  and 
partiality 

•  Train  in  the  concepts  of  Bayesian 
theory 

•  Train  to  estimate  posterior 
probabilities 

PREFERENCE 
SPECIFICATION 
(=  specifying  which  of 
several  outcomes  is 
preferred) 

•  Capable  of  "holistic,"  multidimensional  evaluations 

•  Number  of  dimensions  that  can  be  handled  successfully  is 
limited 

•  Random  error  increases  with  number  of  dimensions  considered 

•  Has  a  tendency  to  rationalize  (to  make  up  the  numbers  a 
posteriori  to  fit  an  implicitly  made  decision) 

■ 

•  Train  people  to  make  judgments 
independent  of  the  worths  of  decision 
outcomes 

•  Train  people  to  make  judgments 
independent  of  worth  measuring 
technique 

•  Train  in  decomposition  and 
aggregation  methods 

•  Train  people  to  make  explicit  and 
prioritize  (weigh)  worth  factors 

ACTION  SELECTION 
(=  the  decision  point) 

•  Must  often  occur  under  high  residual  uncertainty 

•  Selections  made  when  in  a  position  of  disadvantage 
considerably  worse  than  selections  made  in  a  position  of 
advantage 

•  Train  under  conditions  of 
disadvantage 

DECISION 
-  EVALUATION 
(=  evaluating  decision 
"quality") 

•  Not  treated  as  a  process  performed  by  the  decision  maker 
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Table  2.  Human  Decision-Making  Characteristics  of  Boff  and  Lincoln  (1988) 


Type  Dedsion-H 
Makistg  Task 

<^aract&mtic  Bdiayiof 

Source 

Estimates  of 
Descriptive  Statistics 

•  Good  at  estimating  means 

•  Good  at  estimating  proportions  but  shows  some  tendency  to  underestimate 
high  proportions  and  overestimate  low  proportions 

•  Poor  at  estimating  sample  variance;  usually  underestimates  it 

Peterson  &  Beach  (1967) 
Peterson  &  Beach  (1967) 

Peterson  8c  Beach  (1967), 

Sage  (1981),  Schrenk  (1969) 

Statistical  Inferences 
from  Samples 

•  Very  conservative  in  making  probability  estimates,  probably  due  to 
misunderstanding  of  sampling  distributions 

•  Believes  small  samples  to  be  more  typical  of  populations  than  is  warranted 

Peterson  &  Beach  (1967) 

Peterson  &  Beach  (1967) 

Understanding  and 
Use  of  Probability 
Statistics 

•  Good  at  understanding  and  using  probability  statements  not  based  on 
frequency  (e.g.,  "The  probability  of  rain  today  is  0.7") 

•  Tends  to  overestimate  probability  of  favorable  outcomes  and  underestimate 
probability  of  unfavorable  outcomes 

Peterson  &  Beach  (1967), 

Sage  (1981) 

Sage  (1981) 

Problem  Change 
Recognition 

•  Too  conservative  in  recognizing  changes  in  problem  conditions 

•  Delays  too  long  in  response  to  those  changes 

Schrenk  (1969),  Vaughan  & 
Mavor(1972) 

Situation  Diagnosis 

•  Poor  at  making  diagnosis  of  complex  situations  entailing  complicated 
interpretations  of  configural  cue  patterns 

Hopf-Weichel  et  al.  (1979), 
Vaughan  &  Mavor  (1972) 

Formulation  and 
Selection  of  Action 
Alternatives 

•  Not  sufficiently  inventive  and  tends  to  adopt  the  first  solution  developed 

•  Forms  hypotheses  early,  then  tries  to  confirm  rather  than  test  them 

•  Does  not  consider  enough  hypotheses 

Hopf-Weichel  et  al.(1979), 
Vaughan  &  Mavor  (1972) 

Schrenk  (1969) 

Identification  and 

Use  of  Decision 
Criteria 

•  Finds  it  difficult  to  use  more  than  one  or  two  criteria  at  a  time 

•  Tends  to  identify  only  those  criteria  favorable  to  selected  action 

Hopf-Weichel  et  al.  (1979), 
Vaughan  &  Mavor  (1972) 

Use  of  Available 
Information 

•  Tends  to  use  only  concrete,  high-confidence  facts  and  prefers  to  ignore 
ambiguous  or  partial  data 

•  Asks  for  more  data  from  sources  of  good  quality  information 

•  Requests  more  evidence  than  is  necessary  for  a  decision 

•  Poor  at  combining  evidence  to  update  probability  estimates 

•  Gives  undue  weight  to  early  events  and  is  reluctant  to  change  an  erroneous 
commitment  in  light  of  new  evidence 

Hopf-Weichel,  etal.(1979), 
Vaughan  &  Mavor  (1972) 
Schrenk  (1969),  Vaughan  & 
Mavor  (1972) 

Schrenk  (1969) 

Schrenk  (1969) 

Dale  (1968),  Schrenk  (1969) 

Detection  of  Change 
in  Statistical 
Properties  of 
Monitored  Processes 

•  Exhibits  near  optimal  behavior  in  optimal  estimation  and  optimal  control 
experiments,  even  though  sophisticated  interpretation  of  dynamic  probabilities 
is  involved 

The  overall  conclusion  jfrom  this  table  is  that  "as  statistical  decision  makers,  humans  are 
generally  inefficient  at  making  estimates  of  either  descriptive  or  inferential  statistics. 
Conversely,  humans  are  effective  decision  makers  when  they  have  had  input  into  the  situations 
they  are  monitoring,  and  when  they  are  highly  practiced"  (Boff  &  Lincoln,  1988). 

Several  observations  can  be  made  at  this  point: 


1 .  The  two  tables  complement  each  other  and  present  a  somewhat  fragmentary  picture 
of  the  human  decision  maker  as  a  basically  suboptimal  (perhaps  in  rare  cases  near- 
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optimal)  performer  of  aspects  or  subtasks  in  decision  making.  Whether  one  accepts 
that  picture  depends  on  whether  one  believes  the  methodological  threats  to 
generalizability  of  laboratory  results  to  real-world  situations  must  be  taken 
seriously. 

2.  Whether  these  subtasks  are  performed  is  a  function  of  the  nature  of  the  specific 
decision-making  task,  but  there  is  not  a  systematic  classification  of  decision-making 
tasks  based  on  performance  requirements. 

3.  The  subtasks  of  decision  making  (see  the  first  column  of  either  table)  address  the 
uncertainty  reduction  process  that  occurs  between  the  recognition  point  and  the 
decision  point.  They  do  not  provide  any  insight  into  the  preceding  recognition 
process  nor  into  the  succeeding  implementation  and  feedback  processes. 

4.  The  various  subtasks  are  to  some  extent  arbitrary  conjectures  based  on  process 
models  that  arise  firom  logical  considerations  of  what  decision  makers  ought  to  be 
doing  rather  than  fi-om  in-situ  observations  of  real-world  decision  making  that 
would  have  revealed  what  they  are  actually  doing. 

The  prescriptive  focus  can  be  explained  as  the  legacy  of  decision  theory  that  has  so 
dominated  the  thinking  about  the  decision-making  process  that  an  unbiased  descriptive  view  was 
essentially  prevented  for  decades.  This  is  not  to  say  that  prescriptively  oriented  research  did  not 
produce  or  is  not  producing  valuable  results.  The  knowledge  represented  by  Tables  2  and  3  is 
necessary;  it  is  simply  not  sufficient.  If  those  tables  represented  all  the  knowledge  available  for 
developing  guidelines  for  training  design,  training  would  be  matter  of  teaching  prospective 
decision  makers  what  not  to  do  rather  than  what  they  should  do.  Fortimately,  this  is  not  the  case. 
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3.2.3.  Contributions  of  Recent  Research 

Between  1985  and  1988,  Klein  and  Calderwood  (1990)  explored  decision  making  in 
operational  settings  using  a  combination  of  field  studies  and  experiments  to  test  specific 
h5T)otheses.  The  decision  domains  included  in  this  research  were  those  of  urban  fire  ground 
commanders,  wildland  fire  incident  commanders,  and  U.S.  Army  tank  platoon  commanders. 
Their  most  important  findings,  which  were  in  many  cases  at  odds  with  the  traditional, 
prescriptively  biased  concept  of  the  decision-making  process,  added  considerable  depth  to  a 
purely  descriptive  view  of  decision  making.  These  findings  are  as  follows. 

•  Experienced  decision  makers  rely  more  on  situation  assessment,  while  novices 
rely  more  on  option  evaluation  strategies. 

•  Situation  assessment  seems  to  involve  schematic  or  prototypical  knowledge  of 
cues,  goals,  and  expectancies  that  apply  to  a  given  class  of  events.  Current 
cognitive  research  paradigms  have  not  addressed  how  complex  decision 
events  are  classified. 

•  Although  experts  and  novices  notice  the  same  cues  in  a  situation,  novices 
draw  fewer  inferences  based  on  these  cues.  Novices  tend  to  miss  the  tactical 
implications  of  situational  cues. 

•  At  least  in  the  domains  studied  here,  decisions  are  most  likely  to  be  made 
without  any  conscious  deliberation  between  option  alternatives. 

•  When  deliberation  occurs,  decision  makers  are  more  likely  to  use  serial 
evaluation  strategies  than  concurrent  evaluation  of  options.  Serial  strategies 
offer  a  means  of  minimizing  the  calculational  burden,  and  maximizing  the 
speed  with  which  a  decision  can  be  implemented. 

•  Serial  evaluation  is  associated  with  satisfying  rather  than  optimizing  strategies 
and  is  preferred  under  time-pressured  conditions. 

•  Options  are  frequently  evaluated  through  the  use  of  images  or  a  "mental 
model"  that  operates  as  a  simulation  forjudging  whether  an  option  will  be 
successful  in  a  specific  case. 
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•  Expert  decision  makers  rely  on  a  process  of  "progressive  deepening"  or 
reasoning  into  the  future. 

•  Analogical  reasoning  is  infrequently  reported,  which  suggests  that  the 
processes  involved  in  selecting  and  using  analogs  are  relatively  automatic  and 
unconscious. 

•  When  analogs  are  used  (often  to  address  aspects  of  a  problem  that  are  not 
routine),  they  are  critical  to  option  selection.  Thus,  inappropriate  analogs  are 
a  primary  cause  of  errors. 

•  Time  pressure  does  not  affect  the  quality  of  decisions  made  by  experts  as 
much  as  it  affects  novices  because  experts  rely  more  heavily  on  rapid 
recognitional  processes. 

Klein  and  Calderwood  (1990)  have  synthesized  their  findings  in  their  Recognition- 
Primed  Decision  (RPD)  model  (Figure  6). 

Figure  6.  Recognition-Primed  Decision  Model  by  Klein  and  Calderwood  (1990) 
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During  the  same  time  frame,  Noble  and  his  associates  (Noble,  Grosz,  &  Boehm-Davis, 
1987)  conducted  a  series  of  studies  to  examine  the  development  and  use  of  schemata  in  decision 
making.  Starting  with  a  general  model  of  decision  making  proposed  by  Lawson  (1987),  they 
distinguish  two  modes  of  decision  making:  Rational  Outcome  Calculation  (ROC)  and 
Recognition-Based  Decision  Making  (RB).  The  former  is  defined  as  a  "rational  process  of 
explicitly  comparing  options  and  choosing  the  optimal  alternative,"  while  the  latter  is  a  mode 
where  "the  decision  seems  to  follow  directly  from  a  recognition  of  the  type  of  situation  and  a 
recollection  of  what  actions  usually  work  well  in  this  kind  of  situation."  Noble  and  his  associates 
see  these  two  modes  as  the  two  extremes  of  a  continuum  of  decision-making  strategies,  where 
the  space  between  these  poles  is  occupied  by  what  they  call  "hybrid"  strategies.  This  concept  is 
illustrated  in  Figure  7. 


Figure  7.  Continuum  of  Decision-Making  Process  Types 
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The  relationship  between  Klein's  Recognition  Primed  Decision  Making  and  Noble  et  al.’s 
Recognition  Based  Decision  making  is  obvious.  Hammond  (1986)  essentially  makes  the  same 
distinction  when  he  refers  to  "analytic"  versus  "intuitive"  modes  of  decision  making.  A  similar 
distinction  is  also  made  by  Rasmussen  (1988)  who  differentiates  between  rational,  knowledge- 
based  decision  making  and  heuristic,  rule-based  decision  making  and  indicates  that  the  former 
process  is  used  by  novices  (and  by  experts  facing  unfamiliar  situations)  and  that  the  latter  is 
"applied  by  skilled  actors." 
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Particularly  interesting  for  the  purposes  of  this  paper  is  the  Noble  et  al.’s  (1987)  account 
of  what  influences  the  use  of  one  or  the  other  strategy: 

Three  factors  influence  the  relative  role  played  by  recognition  versus  outcome 
calculation  in  a  decision.  These  are  knowledge  or  familiarity  with  decision.  The 
more  familiar  a  person  is  with  a  situation,  the  larger  the  role  subconscious 
classification  will  play  in  his  decision  making.  Difficulty  in  projecting  outcomes 
caused  by  time  or  resource  limitations  or  by  a  large  number  of  situation 
uncertainties  encourages  a  person  to  rely  more  on  the  classification  process. 

Finally,  the  less  important  it  is  that  the  person  make  the  very  best  decision,  the 
more  likely  the  person  is  to  rely  on  subconscious  classification,  (p.7) 

Noble  et  al.  later  state: 

The  extent  to  which  these  two  elements  arise  in  any  decision  will  depend  on  the 
characteristics  of  the  task,  the  familiarity  of  the  decision  maker  with  the  task,  the 
severity  of  consequences  for  bad  decisions,  and  preferred  decision  modes  specific 
to  the  individual  decision  maker,  (p.  9) 

Noble  et  al.  (1987)  concluded  that  schemata  are  the  foundation  for  expert  decision 
making.  Although  they  cannot  offer  any  insight  into  how  such  schemata  are  developed,  they  are 
assumed  to  be  a  product  of  experience.  The  schemata  allow  decision  makers  to  identify  high 
quality  alternatives  directly  from  the  situation.  Further,  they  are  organized  around  prototypes 
and  contain  feature  data  that  enable  people  to  evaluate  the  significance  of  differences  between  an 
observed  situation  and  the  typical  prototype  situation. 


The  results  obtained  by  Klein  et  al.  (1990)  and  Noble  et  al.  (1987)  are  consistent  with 
each  other  and  with  the  findings  of  much  of  the  research  dealing  with  the  nature  of  expertise  and 
the  differences  between  novices  and  experts.  The  picture  of  the  human  decision-making  process 
that  arises  from  the  combined  results  of  this  research  can  be  summarized  as  follows. 

1 .  There  are  two  distinct  pure  forms  of  the  decision-making  process  that  rest  at  opposite 
ends  of  a  continuum  populated  by  mixed  or  "hybrid"  processes.  One  pure  form  is  the 
ROC  process,  which  involves  conscious,  analytical,  knowledge-based  processing 
within  an  overall  logical,  systematic  framework.  The  other  pure  form  is  the  RB 
process,  which  involves  unconscious,  intuitive,  heuristic-based  processing. 
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2.  How  much  either  process  is  used  on  a  given  occasion  is  a  function  of  decision  task 
characteristics  and  decision  maker  characteristics.  Keeping  task  characteristics 
constant,  novice  decision  makers  will  operate  closer  to  the  ROC  end  of  the  continuum 
and  experts  will  operate  closer  to  the  RB  end  of  the  continuum.  Keeping  experience 
constant,  decreasing  time  pressure  and  increasing  task  complexity,  increasing  task 
risk  (consequences  of  bad  decisions),  and  increasing  imcertainty  will  increase  the  use 
of  ROC  processes  over  RB  processes  and  vice  versa. 

3.  The  ROC  process  is  a  weak,  general,  rational  problem-solving  procedure  that  is 
probably  minimally  adapted  to  the  gross  features  of  the  decision-making  domain  and 
is  to  some  extent  modified  by  prior  experience  in  related  or  similar  domains.  The 
initial  concern  is  reduction  of  situation  uncertainty  which,  in  Nickerson  and  Feehrer's 
(1975)  terms,  probably  involves  processes  of  information  gathering,  data  evaluation, 
problem  structuring,  hypothesis  generation,  and  hypothesis  testing.  The  second 
concern  is  the  reduction  of  uncertainty  regarding  goals  to  be  achieved  by  the  decision. 
This  probably  involves  explicit  clarification  of  personal  goals,  consultation  of 
external  "public"  sources  of  guidance,  or  both.  The  third  and  really  central  concern  is 
the  reduction  of  uncertainty  concerning  options.  This  reduction  will  involve 
generation  of  the  option  set,  assessment  of  the  possible  option  effects  in  view  of  the 
goals  to  be  achieved,  and  finally  an  assessment  of  the  feasibility  of  each  option.  The 
ROC  process  will  generally  be  performed  with  the  intent  of  achieving  a  decision  that 
is  logically  consistent  with  the  available  information  and  optimal  as  far  as  goal 
achievement  and  feasibility  is  concerned. 

4.  The  RB  process  is  characterized  by  rapid  or  near  instantaneous  and  not  necessarily 
conscious  aggregation  of  situational  data  into  a  situational  assessment.  The  situation 
is  identified  as  a  member  of  a  problem  class  that  is  defined  by  a  prototype  (which  is 
probably  a  schema).  Simultaneous  with  situation  recognition,  the  prototypical 
solution  (which  is  also  probably  a  schema)  appears  for  members  of  this  situation 
class.  Deviations  of  the  actual  situation  firom  the  prototype  situation  are  recognized 
and  provide  cues  for  adapting  the  prototypical  solution  to  the  current  actual  situation. 
In  general,  experts  first  are  concerned  with  the  reduction  of  situation  imcertainty. 

This  initial  problem  solved,  they  "see"  the  problem  md  a  fairly  specific  solution  for  it 
at  the  same  time,  then  spend  the  remainder  of  the  available  decision  time  primarily 
reducing  uncertainty  associated  with  the  feasibility  of  the  one  option  they  are 
considering.  The  primary  mechanism  for  reducing  this  type  of  uncertainty  appears  to 
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be  something  like  imagery-based  simulation  using  some  sort  of  "runnable"  mental 
model.  Uncertainty  concerning  goals  to  be  achieved  by  the  decision  commonly  does 
not  appear  to  be  a  concern;  that  is,  experts  are  familiar  with  the  general  goals  of  their 
job  and  "know"  what  goals  can  or  cannot  be  achieved  in  a  specific  situation.  The  RB 
process  is  generally  performed  with  the  intent  of  satisfying  goals  and  requirements 
rather  than  optimizing  their  achievement. 

This  is  certainly  a  richer,  more  comprehensive,  and  more  realistic  view  of  the  decision¬ 
making  process  than  that  produced  by  the  earlier,  prescriptively  contaminated  research; 
howeaver  it  is  not  so  much  an  alternative  view  as  it  is  a  complementary  one.  The  concept  of 
option  assessment  and  choice,  which  was  central  to  earlier  views  of  decision  making,  is 
confirmed  as  a  feature  of  the  process  engaged  in  by  novices.  It  is  also  central  when  the  task  is 
very  important  and  when  there  is  ample  time  to  perform  it.  But  when  experts  are  performing 
under  time  pressure,  its  centrality  evaporates  and  gives  way  to  concerns  of  situation  recognition 
and  feasibility  assessment.  The  biases  and  weaknesses  of  humans  as  statisticians  and  probability 
estimators  are  probably  afflictions  that  have  their  greatest  effect  when  decision  making  proceeds 
along  the  lines  of  the  ROC  process;  it  is  in  this  process  that  knowledge  of  biases  and  weaknesses 
might  be  usefully  brought  to  bear.  On  the  other  hand,  rigidity,  lack  of  creativity  in  problem 
structuring,  qnd  perhaps  perseverance  are  probably  dangers  that  experts  should  guard  against. 
Their  ingrained  schemata  could  prove  dysfunctional  in  situations  that  are  superficially  close  to 
typical  situations. 

Although  this  is  a  richer  and  more  comprehensive  description,  the  process  is  still  not 
described  in  its  entirety.  Several  issues  have  not  been  addressed  yet,  such  as  how  a  decision 
maker  recognizes  the  need  for  a  decision  in  the  first  place  and  with  what  factors  influence  the 
duration  arid  reliability  of  the  recognition  process.  There  also  has  been  little  concern  with  the 
"aftermath"  —  the  processes  that  occur  after  the  decision  is  made  (after  the  DP).  For  example, 
very  little  is  known  about  how  the  "embedded"  decision  on  when  to  start  implementation  is 
influenced  by  task  or  skill  factors,  how  feedback  is  used  during  or  after  implementation,  what 
constitutes  useful  feedback,  and  how  feedback  is  employed  in  decision  chains  as  opposed  to 
singular,  independent  decisions.  However,  the  crucial  mid-phase,  which  encompasses  what  is 
more  narrowly  xmderstood  as  decision  making,  has  become  much  better  imderstood. 
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3.2.4.  Process  Models 


One  simple  and  illuminating  way  to  model  the  mid-phase  is  offered  by  the  inference 
structure  of  heuristic  classification  in  Clancey  (1985)  (Figure  8).  The  data  abstractions  are,  in 
fact,  situation  classes  represented  by  prototype  schemata.  The  solution  abstractions  are  option 
classes  represented  by  schemata  of  prototypical  solutions.  Situation  and  option  classes  are  linked 
by  a  heuristic  match.  The  option  classes  are  further  refined  into  specific  options.  Both  experts 
and  novices  traverse  this  arc  of  an  inference  structure  fi-om  bottom  left  to  bottom  right,  with 
novices  slogging  through  it  laboriously,  analytical  step  by  analytical  step,  while  the  experts  see 
the  entire  pattern  in  one  fell  swoop,  perhaps  dimly  in  places  at  first  but  with  increasing  clarity  as 
they  probe  here  and  there  for  missing  or  confirming  elements  of  information. 


Figure  8.  Clancey’s  (1985)  Inference  Structure  of  Heuristic  Classification 
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There  are  striking  parallels  between  Clancey's  (1985)  model  and  a  model  of  the  decision¬ 
making  process  developed  by  Rasmussen  (1988).  There,  too,  a  set  of  concrete  initial  data  leads 
to  the  identification  of  a  situation  class  that  is  linked  to  a  desired  goal  state  which  is  then  refined 
into  an  executable,  concrete  solution.  Last,  but  not  least,  we  suggest  a  process  model  of  decision 
making  that  is  based  on  the  uncertainty  and  time  line  models  presented  in  the  preceding  section. 
The  process  model  focuses  on  the  uncertainty  reduction  process  and  covers  the  cognitive 
activities  between  the  RP  and  DP.  This  model  (Figure  9)  is  not  descriptive.  At  best,  it  is  a 
descriptive  hypothesis:  Given  the  logical  relationships  between  types  of  uncertainty,  this  is  the 
sequence  decision  makers  can  be  expected  to  follow.  Whether  they  actually  do  it  this  way  has 
not  been  shown  empirically.  One  may  also  hypothesize  that  the  model  is  a  useful  prescriptive 
"job  aid"  for  decision  making.  Whether  that  is  true  will  be  one  issue  that  can  be  tested 
empirically  with  the  system  to  be  developed  under  this  project. 
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Figure  9.  Process  Model  for  Decision  Making 
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The  two  key  notions  embedded  in  this  model  are  (1)  the  notion  of  logical  order  of 
precedence  in  uncertainty  reduction  and  (2)  the  notion  that  goals,  option  classes,  and  specific 
options  form  a  means-ends  tree.  Logically,  uncertainty  reduction  should  follow  the  order 
Situation  -  Goals  -  Options:  One  cannot  make  much  headway  with  either  goals  or  options  until 
the  situation  has  been  clarified,  and  one  cannot  weigh  options  until  the  goals  have  been  clarified. 

A  particularly  interesting  issue  is  the  question  of  what  happens  in  the  box  labeled  "Edit, 
refine  and  evaluate  the  options."  The  prescriptive  view  as  expressed  by  the  multi-attribute 
theory,  Bayesian  statistics,  or  Fogel's  (1992)  valuated  state  space  is  not  representative  of  what 
decision  makers  actually  do.  On  the  other  hand,  a  validated  descriptive  view  for  the  set  of 
cognitive  activities  lumped  together  in  this  box  does  not  yet  exist. 

It  is  surmised  that  decision  makers  construct  the  kind  of  relatively  simple  means-ends 
trees  shown  in  Figure  10  with  a  goal  set  on  top,  option  classes  in  the  middle,  and  specific  options 
at  the  bottom.  It  is  also  surmised  that  tree  construction  and  editing  is  a  fluid  process,  where 
perhaps  only  the  goal  set  is  relatively  stable,  and  option  classes  and  specific  options  are  added, 
subtracted,  refined,  and  evaluated  as  new  information  becomes  available  (i.e.,  as  the  decision 
maker  progresses  in  efforts  to  reduce  uncertainty). 

Figure  10.  Means-Ends  Tree  Concept 
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This  tree-building  and  editing  activity  probably  changes  considerably  as  expertise 
increases.  Novices  can  only  develop  a  few  options  or  at  most  a  rather  sparse  tree  because  they 
simply  do  not  know  enough  to  do  more.-  As  domain  knowledge  and  experience  increases,  the 
tree  is  likely  to  become  larger  and  more  complete  up  to  a  point,  then  the  tree  begins  to  "shrink" 
again.  The  expert  has  a  very  sparse  tree  where  a  goal  set,  an  option  class,  and  a  single  option 
form  one  clear,  coherent  path. 


29 


This  view  provides  one  possible  explanation  for  the  differences  between  novice  and 
expert  performance:  The  expert  has  seen  the  tree  before  and  can  very  quickly  prune  it  down  to 
the  most  promising  option  class  and  specific  option,  whereas  the  novice  must  develop  and  assess 
the  entire  tree  branch  by  branch.  This  leads  to  a  preoccupation  with  option  generation  by  novices 
who  use  the  ROC  process  and  to  a  preoccupation  with  feasibility  of  specific  options  by  experts 
who  are  running  the  RB-type  of  process. 

As  decision  makers  work  on  reducing  various  categories  of  uncertainty,  they  acquire  new 
information  that  allows  them  to  prune  or  expand  the  tree,  edit  option  classes  and  options,  and 
evaluate  and  compare  options.  The  latter  activity  is  necessarily  concerned  with  option  effects 
and  option  feasibility.  Options  that  have  a  high  potential  to  satisfy  the  goal  set  and  a  high 
potential  of  being  feasible  are  of  course  more  valuable  than  options  that  have  low  effects  and 
feasibility  potentials.  But  evaluation  must  also  be  concerned  with  uncertainty,  as  illustrated  in 
Figure  1 1 .  An  option  with  a  high  but  very  uncertain  effects  potential  (Option  2)  is  less  valuable 
than  an  option  with  equally  high  effects  potential  but  less  effects  \mcertainty  (Option  1);  of 
course,  the  same  is  true  for  the  feasibility  variable.  Figure  1 1  represents  a  simple  option 
evaluation  model  that  has  the  potential  of  being  descriptive  because  of  its  relative  simplicity. 

Figure  1 1 .  Simple  Option  Evaluation  Model 
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3.2.5.  Training  Considerations 

Three  major  points  can  be  derived  from  the  preceding  discussion.  The  first  is  the  notion 
of  human  deficiencies  in  decision  making.  Research  in  this  area  continues  to  be  actively  pursued 
today.  For  example,  Tolcott,  Marvin,  and  Lehner  (1989)  are  investigating  bias  phenomena  in  the 
treatment  of  confirming  versus  disconfirming  evidence.  This  is  also  the  area  where  Nickerson 
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and  Feehrer's  1975  report  provides  its  most  explicit  and  insightful  training  guidance.  Their 
thoughts  on  that  subject  and  its  training  implications  are  still  valid  today  and  can  be  siimmarized 
as  follow. 

•  There  are  two  types  of  human  deficiencies;  Nickerson  and  Feehrer  label  these 
"deficiencies"  and  "limitations."  Deficiencies,  such  as  the  tendency  to  be  overly 
conservative  in  the  use  of  probabilistic  information,  can  presumably  be  corrected  or 
"trained  out."  Limitations,  such  as  "the  inability  of  most  people  to  weigh  more  than 
some  small  number  of  factors"  cannot  be  corrected;  they  can  only  be  "trained  around" 
or  helped  by  job  aids. 

•  If  experimental  research  can  reliably  identify  some  fairly  universal  deficiencies  (and, 
given  the  methodological  problems  of  such  research  this  is  a  big  if  indeed),  one  would 
obviously  want  to  correct  these  deficiencies  during  training.  Nickerson  and  Feehrer 
suggest  exposing  students  to  a  wide  variety  of  decision-making  situations  in  which  a 
particular  deficiency  is  likely  to  appear,  then  providing  immediate  feedback  to  the 
students  on  the  appropriateness  of  their  behavior. 

•  With  regard  to  limitations,  Nickerson  and  Feehrer  suggest  that  "the  goal  should  be  to 
educate  the  decision  maker  concerning  what  those  limitations  are  and  to  provide  him 
with  the  means  for  working  around  them."  In  other  words,  these  authors  suggest  a 
combination  of  consciousness  raising  and  job  aiding. 

The  researchers  involved  in  this  current  effort  concur  with  these  suggestions  and  advocate 
an  approach  that  begins  with  a  careful  selection  of  a  few  deficiencies  and  limitations.  Prime 
candidates  are  those  that  are  both  reliably  identified  by  a  variety  of  research  paradigms  and  likely 
to  have  highly  adverse  effects  on  decision-making  results.  The  selected  deficiencies  should  then 
be  "trained  out"  through  explanation,  exposure,  and  feedback.  The  selected  limitations  should  be 
"trained  around"  through  explanation  and  by  providing  methods  or  procedures  that  assist  learners 
in  coping  with  information  processing  demands  for  which  they  are  ill-equipped.  These 
recommendations  are  summarized  in  the  following  two  guidelines: 


Instructional  Design  Guideline  No.  5 

&  Train  students  how  to  avoid  and/or  overcome  deficiencies  by  using  practice  situations  in 
which  deficiencies  are  likely  to  show  up. 
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Instructional  Design  Guideline  No.  6 

I&  Train  students  to  cope  with  limitations  by  including  explanations,  procedural  aids,  and 
appropriate  practice  and  feedback  in  situations  where  limitations  are  likely  to  manifest 
themselves. 


The  second  major  point  derived  from  the  discussion  of  process  models  is  that  novices  and 
experts  engage  in  qualitatively  and  quantitatively  different  processes  or  modes  during  decision 
making.  How,  then,  might  one  optimally  promote  the  development  of  an  RB  mode  through 
training?  One  obvious  strategy  is  to  provide  ample  amounts  of  practice  in  realistic  and 
increasingly  complex  decision-making  situations.  However,  there  is  the  potential  to  do  more. 
Because  the  ROC  and  RB  modes  are  vastly  different,  instructional  treatments  that  are  optimal  for 
a  novice  engaging  in  ROC  decision  making  are  not  necessarily  optimal  at  higher  levels  of 
proficiency  where  the  learner  begins  to  abandon  the  ROC  mode  and  to  engage  in  the  RB  type  of 
decision  making  (i.e.,  instructional  treatments  are  likely  to  be  more  effective  if  they  are  adapted 
to  the  learner's  level  of  proficiency).  For  example,  a  learner  at  the  early  ROC  stage  might  benefit 
from  being  held  to  a  systematic,  rational,  and  explicit  approach.  As  the  learner's  proficiency 
increases,  explicit  cues  and  procedures  can  be  abbreviated  and/or  replaced  by  "compiled"  or 
condensed  cues.  Finally,  such  cues  can  be  eliminated  completely.  This  notion  of  adapting  the 
instructional  treatment  to  learner  proficiency  is  explored  further  in  the  next  section,  where  the 
focus  is  on  the  learning  process  for  decision-making  skills.  At  this  point,  we  suggest  the 
following  guidelines: 


Instructional  Design  Guideline  No.  7 

&  Provide  high  levels  of  increasingly  complex  practice  in  realistic  decision-making  situations. 


Instructional  Design  Guideline  No.  8 

&  Adapt  instructional  treatments  to  learner  proficiency  by  gradually  compiling  and  finally 
withdrawing  instructional  cues. 


The  third  and  final  significant  point  is  the  notion  of  a  general  or  xmiversal  method  for 
decision  making  that  goes  along  with  the  imcertainty  and  the  time  line  models  developed  in 
Section  3.1 .  This  model,  while  logically  valid,  has  not  yet  been  proven  as  a  valid  description  of 
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either  ROC  or  RB  inodes  in  decision  making.  However,  it  might  be  a  valid  description  of  either 
mode,  and  it  might  function  as  a  useful  methodological  prescription  for  decision  makers  at  any 
level  of  proficiency.  Whether  this  is  right  is  an  issue  for  empirical  research.  Since  this  project 
aims  to  produce  a  training  system  that  can  be  used  for  research,  the  following  guideline  is 
offered: 


Instructional  Design  Guideline  No.  9 

I&  Include  in  the  training  system  the  capability  to  train  students  in  using  the  process  model  for 
decision  making. 


3.3.  How  Do  People  Learn  Decision  Making? 

3.3.1.  Introduction 

How  do  people  leam  the  decision-making  process?  The  ideal  answer,  again,  is  a 
comprehensive  descriptive  model  that  accounts  for  the  entire  learning  process  from  the  state 
represented  by  novices  in  a  particular  domain  to  the  state  represented  by  domain  experts.  The 
volume  of  existing  theoretical  and  empirical  research  that  could  be  brought  to  bear  on  that  issue 
is  exceedingly  large.  A  comprehensive  and  thorough  review  of  this  body  of  literature  is  simply 
impossible  under  the  constraints  of  this  project,  even  if  the  focus  of  the  review  is  restricted  to  the 
subset  of  research  that  deals  with  the  acquisition  of  complex  cognitive  skills.  Therefore  the  focus 
has  been  narrowed  and  restricted  to  a  number  of  independently  developed,  multistage  skill- 
acquisition  theories.  The  notion  of  multiple  stages  in  the  learning  process  is  interesting  from  the 
viewpoint  of  instructional  design  because  the  existence  of  stages  might  allow  precise  tailoring  of 
instructional  treatments  to  each  stage,  which  in  turn  might  lead  to  faster,  more  efficient  learning. 
This  notion  was  introduced  in  the  preceding  section  where  the  focus  was  on  decision-making 
performance;  here,  this  notion  will  be  expanded  further. 

The  literature  on  novice-expert  differences  gives  an  idea  of  the  territory  that  must  be 
traversed  during  the  learning  process.  If  there  are  indeed  identifiable  stages  in  the  process,  they 
can  serve  as  guideposts  for  the  journey  through  what  otherwise  is  "terra  incognita."  However,  in 
negotiating  these  stages,  one  will  likely  find  that  a  singular  type  of  instructional  treatment  cannot 
be  effective  and/or  efficient  throughout  the  entire  process.  Consequently,  the  instructional 
treatment  must,  in  some  fashion,  adapt  to  the  changes  that  occur  in  the  learner  in  order  to  achieve 
optimal  promotion  of  the  learning  process.  That  adaptation  would  have  to  take  its  cues  from 
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identifiable  changes  in  learner  behavior.  For  example,  a  multistage  skill  acquisition  theory  can 
achieve  practical  significance  for  instructional  design  if  it  can  demonstrate  empirically  that  the 
stages  are  clearly  and  easily  detectable  and  if  it  offers  insights  that  can  be  translated  into 
instructional  design  prescriptions  that,  when  correctly  implemented,  will  promote  transition  from 
one  stage  to  the  next. 

3.3.2.  Multistage  Skill  Acquisition  Models 

A  number  of  multistage  theories  were  examined  in  the  course  of  this  project.  The 
common  themes  running  through  these  theories  led  to  the  development  of  a  Unified  Model  for 
Skill  Acquisition  (UMSA)  —  a  synthesis  of  several  multistage  skill  acquisition  theories.  The 
theories  examined  for  the  UMSA  and  the  stages  proposed  by  each  theory  are  presented  in  Table 
3.  These  theories  describe  stages  in  terms  of  changes  along  a  number  of  dimensions. 
Commonalities  were  foimd,  both  in  the  dimensions  that  were  used  and  in  the  changes  that  were 
described  for  these  dimensions.  In  other  words,  the  commonalities  went  beyond  the  common 
theme  of  a  gradual  process  marked  by  stages.  It  was  therefore  believed  that  one  could,  without 
excessive  distortions,  fuse  the  essential  themes  in  each  theory  into  one  common,  imified  model 
that  would  come  closer  to  descriptive  truth  than  did  each  constituent  models  alone.  The  result 
was  the  UMSA  (Table  4)  in  a  form  slightly  abbreviated  from  the  original. 


Table  3.  Multistage  Skill  Acquisition  Theories  and  Their  Stages 


Theories 

Stages  1 

Dreyfus  &  Dreyfus 
(1980) 

Novice 

Advanced 

Competent 

Proficient 

Expert 

Siegler(1978) 

Rule  I  Behavior 

Rule  II  Behavior 

Rule  III  Behavior 

Rule  rv  Behavior 

Gentner  (1980); 
Forbus  &  Gentner 
(1986) 

Protohistories 

Causal  Corpus 

Naive  Physics 

Expert  Models 

McDermott  & 

Larkin  (1978) 

Verbal 

Naive 

Scientific 

Mathematical 

Anderson  (1982) 

Cognitive 

Compilation 

Procedural 

Fitts  &  Posner 
(1967) 

Cognitive 

Associative 

Autonomous 

Rasmussen  (1986) 

Knowledge-Based 

Rule-Based 

Skill-Based 
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Table  4.  Unified  Model  for  Skill  Acquisition  (UMSA) 


Level  1 

Stage  1 

Level  i 
St^e2 

Level  2 

Stage  3 

Level  2 

Stage  4 

Level  3 

Stage  5 

• :  •  Level  3 
•  Stage  6 

Novice 

Experienced 

Novice 

Capable 

Learner 

Proficient 

Learner 

Skilled  ' 
Learner 

Expert 

Type  of  Knowledge 

Declarative 

Declarative 

Declarative  with 
simple 

procedures  in 

step-by-step 

format 

Integrated  rules  (2 
to  3  rules) 

Procedural 

Procedural 

Type  of  Learning 

Verbal 

information 

Verbal 
information; 
discrimination 
learning  with 
simple  concrete 
concepts 

Concrete 
concept 
learning;  some 
very  general 
problem-solving 
strategies 

Abstract  concept 
learning;  some 
domain-specific 
problem-solving 
strategies 

High-order  rule- 

learning, 

complex 

problem-solving, 

cognitive 

strategies 

learning 

High-order  rule 

learning, 

complex 

problem 

solving,  more 

general 

cognitive 

strategies 

applied 

Cognitive  Processes 

Memorizing 
facts  and  rules; 
rehearsing 
information  for 
recall 

i 

Comprehending 
facts  and  rules 
relative  to 
context 

Classifying 
based  on 
concrete  fea¬ 
tures;  simple 
links  between 
declarative 
knowledge  & 
procedures; 
simple 

hypotheses  for 
problem 
solution;  some 
meta-cognitive 
processing 

Forming  higher 
order  rules  by 
combining  rules; 
forming  principle- 
based  relalational 
links  in  networks; 
testing  alternative 
hypotheses;  meta- 
cognitive 
monitoring  of 
learning  & 
effectiveness 

Formulating 

high-level 

jproductions; 
elaborative 
inferencing  to 
delimit  possible 
solution  paths; 
active  and 
effective  meta- 
cognitive 
processing 

j 

Refining 
representations 
and  associated 
productions; 
testing 

productions  for 
efficiency  & 
comprehen¬ 
siveness;  active 
and  effective 
meta-cognitive 
processing 

Knowledge 

Organization 

1  Listings  of  facts 
and  rules; 
unrelated 

1  knowledge 
forms 

Simple 

structures  linked 
by  a  few  surface 
features 

More  complex 
and  meaningful 
structures; 
complete  linking 
of  surface 
features 

Modified 
representations 
:  with  principles 
included,  some 
procedural 
relationships 

More  abstract 

representations 

including 

principles  and 

procedures; 

productions  for 

executing 

knowledge 

structures 

Highly 
i  integrated 

1  systems  of 
concepts, 
principles, 
productions, 
and  macro¬ 
productions 

Role  of  Context 

Context-free; 
defined  rules 
detached  from 
total  context 

Some  context- 
specific  features 
with  context-free 
rules 

Increased  focus 
on  context 
composition  & 
similarity  with 
situations  from 
same  domain 

More  attention  to 
;  processes  & 
principles 
underlying  context 
&  their 

generalization  to 
other  settings 

Context  features 
are  related  to 
other  domains; 
context  rules  are 
inferred  from 
varied  settings 

Context- 
specific  rules 
are  abstracted 
into  generalized 
productions; 
automated 
systems  of 
context-free 
and  context- 
specific  rules 
for  particular 
problem  types 

Processing 

Characteristics 

Very  slow;  very 

resource 

demanding 

Very  slow, 
effortful 

Improved 
efficiency  in 
retrieval  of 
declarative 
knowledge  but 
still  slow  and 
effortful 

Improved 
processing  of 
declarative 
knowledge;  slow 
and  consciously 
controlled 
execution 

Much  faster; 
some 

consistency  in 

automated 

productions; 

some  parallel 

processing  with 

familiar 

productions 

Very  fast; 
automatic 
executions; 
parallel 
processing  of 
complex  tasks 
in  the  domain 
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This  model  can  be  criticized  from  a  number  of  perspectives:  the  language  is  vague  and 
jargon-laden,  the  use  of  levels  and  stages  is  confusing  and  arbitrary,  the  lack  of  clear  distinction 
between  some  dimensions  (e.g.,  "type  of  learning"  and  "cognitive  processes")  invites 
speculations  of  reducing  the  number  of  dimensions,  and  the  arbitrariness  in  declaring  six  stages 
(why  not  3  or  4  or  5  or  12?)  suggests  fewer  stages  might  be  just  as  "good."  Also,  there  is  a  real 
question  whether  one  can  legitimately  call  either  Fitts  and  Posner's  (1967)  original  theory  or 
Anderson's  (1982)  extension  of  it  multistage  theories. 

The  key  question  is  whether  this  model  is  useful  for  the  development  of  instructional 
design  guidance.  The  model  was  considered  only  marginally  useful  in  this  form;  however,  we 
are  convinced  of  the  basic  validity  of  a  multistage  model  and  are  intrigued  by  the  potential  of 
such  a  model  to  guide  instructional  design  into  a  greater  adaptiveness  over  the  course  of  skill 
acquisition.  To  be  both  theoretically  convincing  and  practically  useful  for  our  purposes,  such  a 
model  must  be  more  parsimonious,  speak  more  directly  to  the  acquisition  of  decision-making 
skills,  and  give  reasonable  assurance  that  the  defined  stages  are  indeed  identifiable  by  means  of 
appropriate  tests  during  instruction.  Such  a  model  is  presented  below. 

3.3.3.  Learning  Decision-Making  Skills:  A  Simplified  Multistage  Model 

The  simplified  multistage  learning  model  presented  in  Figure  12  is  essentially  a  synthesis 
of  two  concepts:  the  concept  of  a  prerequisite  order  in  the  learning  process  and  the  concept  of 
multiple  stages.  These  two  concepts  must  be  logically  related.  The  prerequisite  concept  holds 
that  a  particular  skill  can  be  learned  only  after  its  prerequisite  skills  have  been  learned.  The 
multistage  concept  also  represents  that  notion;  however,  it  goes  beyond  the  prerequisite  idea  by 
postulating  qualitative  and  quantitative  changes  not  just  in  the  type  of  information  that  can  be 
processed  but  in  how  it  is  processed  (e.g.,  processing  changes  from  serial  to  parallel;  from 
conscious  and  analytical  control  to  unconscious,  intuitive  control;  and  from  very  laggardly  and 
laborious  to  very  fast  and  facile). 

The  prerequisite  notion  has  been  captured  in  a  widely  accepted  two-dimensional  matrix 
by  Merrill  (1983,  1978).  This  matrix  relates  a  "type  of  content"  dimension  to  a  "level  of 
performance"  dimension  and  consists,  for  the  sake  of  operational  utility,  of  ten  discrete  boxes. 
This  matrix  is  usually  viewed  strictly  as  a  convenient  classification  scheme.  It  can  also  be  seen 
as  a  map  of  intellectual  skills  that  must  be  learned  for  some  terminal  objective  (i.e.,  as  something 
like  a  "learning  surface").  When  viewed  from  this  perspective,  the  boxes  are  regions  populated 
with  concepts,  procedures,  and  the  like.  The  boxes  are,  of  course,  simplifying  constructs  with 
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somewhat  fuzzy  borderlines  that  make  it  practically  and  theoretically  easier  to  associate 
particular  instructional  treatments  with  particular  intersections  of  content  type  and  performance 
level. 

Figure  12.  Simplified  Multistage  Learning  Model 


EXPERT 


Implicit  in  this  learning  surface  is  the  notion  that  it  has  to  be  traversed  along  a  central 
diagonal  vector  that  originates  in  the  remember-facts  cell  and  points  towards  the  find-principle 
cell.  That  is  learners  cannot  acquire  a  concept  on  the  remember  level  unless  they  first  acquire  its 
constituent  facts,  cannot  learn  to  use  a  concept  unless  they  first  acquir  the  concept  on  the 
remember  level,  and  so  forth.  This  central  vector  leads,  in  other  words,  from  simple,  disjointed 
knowledge  structures  to  increasingly  complex  and  integrated  knowledge  structures  and  from 
surface  knowledge  to  deep  knowledge. 

The  key  to  the  learning  model,  shown  in  Figure  12,  is  the  notion  that  stages  in  learning 
must  be  defined  as  partitions  of  the  learning  surface.  These  partitions  are  not  sets  of  cells  but 
bands  of  skill  aggregates  that  run  orthogonal  to  the  central  vector  of  the  learning  surface.  Within 
these  bands  and  within  the  cells  included  in  these  bands,  one  can  further  postulate  a  complexity 
dimension  that  runs  parallel  to  and  in  the  same  direction  as  the  central  vector.  This  dimension 
simply  indicates  that  simple  skills  must  be  mastered  before  more  complex  ones.  Finally,  the 
performance-content  matrix  does  not  include  what  is  generally  known  as  expertise.  We  concur 
with  Dreyfus  and  Dreyfus  (1980)  that  the  expert  "no  longer  needs  principles"  and  assert  that 
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expertise  is  in  its  own  category,  its  own  box,  which  is  attached  to  the  upper  right-hand  comer  of 
Merrill's  (1983)  matrix. 


Based  on  these  ideas  and  the  concepts  represented  in  the  earlier  UMSA,  the  learning 
process  for  decision-making  skills  can  be  divided  into  four  stages.  These  stages  synthesize  the 
prerequisite  notion  with  the  multistage  notion.  They  are,  like  the  cells  of  Merrill's  matrix, 
simplifying  constracts  which  should  allow  us  to  associate  instructional  design  guidelines  with 
them.  The  stages  represent  levels  of  decision-making  skill  that  are  achievable  by  learners  if  they 
have  learned  or  are  learning  certain  types  of  content  to  certain  levels  of  performance.  In  general, 
as  learners  proceed  through  these  stages  in  the  direction  of  the  central  vector,  knowledge 
stractures  become  increasingly  complex  and  integrated,  and  knowledge  processing  changes  from 
a  slow,  serial,  analytical,  rational  outcome  calculation  mode  to  a  fast,  parallel,  intuitive, 
recognition-based  mode.  Specifically,  decision-making  performance  on  each  of  the  four  stages 
can  be  characterized  as  follows. 

•  At  the  NOVICE  stage,  decision-making  performance  is  constrained  by  the  lack  of 
domain-specific  knowledge.  The  novice  is  able  to  solve  simple  decision  problems  by 
applying  weak  general  methods  to  a  domain  where  they  are  just  beginning  to  acquire 
some  surface  knowledge  and  where  they  are  best  able  to  use  some  of  the  simpler 
concepts  of  the  domain.  Novices  are  slow  and  unreliable  in  recognizing  decision 
problems  and  cannot  judge  the  available  decision  window.  Their  ability  to 
discriminate  between  salient  and  nonsalient  features  of  the  situation  or  context  is 
rudimentary.  Novices  cannot  prioritize  uncertainty  reduction  requirements  and  have 
no  domain-specific  strategy  for  uncertainty  reduction.  They  generate  incomplete 
option  sets  and,  as  yet,  have  no  concept  of  hierarchical  classes  of  options  (i.e.,  the 
option  sets  are  not  ordered  in  any  sense).  Work  is  performed  exclusively  in  an  ROC 
mode. 

•  At  the  ADVANCED  stage,  decision-making  performance  centers  on  the  use- 
procedures  cell.  Learners  are  now  more  proficient  in  using,  finding,  and  defining 
concepts.  Advanced  learners  begin  to  invent  ways  of  doing  things  (find-procedure 
cell)  and,  develop  a  surface  understanding  of  the  principles  governing  the  domain 
(remember-principle  cell).  Recognition  of  decision  problems  is  timely  and  reliable, 
and  judgment  of  available  decision  time  is  usually  correct.  Learners  begin  to 
recognize  and  utilize  salient  situation  or  context  features,  and  they  can  solve 
moderately  complex  decision-making  problems  in  a  manner  where  weak  general 
methods  are  increasingly  supplemented  and/or  replaced  by  low-level,  domain-specific 
methods.  Learners  begin  to  prioritize  uncertainty  reduction  requirements  and  to  use 
low-level,  domain-specific  uncertainty  reduction  strategies.  Complete  option  sets  are 
identified  as  well  as  major  classes  of  options.  Work  is  still  predominantly  in  an  ROC 
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mode,  but  there  are  instances  of  the  RB  mode  being  employed  with  very  simple  and 
very  frequent  decision  problems. 

•  At  the  COMPETENT  stage,  learners  have  become  reliable  performers.  Competent 
learners  are  still  putting  the  finishing  touches  on  their  ability  to  use  principles  and 
they  are  achieving  competence  in  inventing/finding  procedures  or  rules.  They  now 
are  able  to  focus  on  situational  or  context  features  that  are  germane  to  the  problem 
and  to  modify  options  to  fit  these  features.  Decision  problems  are  recognized  quickly 
and  reliably,  with  knowledge  of  exactly  how  much  time  is  available  and  how  to 
manage  time  well.  Learners  develop  well-ordered  and  complete  option  sets  and 
become  efficient  in  pruning  branches  or  classes  with  low  feasibility,  undesirable 
effects,  excessive  vmcertainty,  and  so  forth,  early  on.  Complex  decision-making 
problems  are  solved  using  domain-specific  strategies  to  reduce  uncertainty.  The 
predominant  working  mode  is  still  the  ROC  mode,  but  instances  of  RB  decision 
making  are  becoming  more  frequent,  especially  with  simple  and  moderately  complex 
decision  problems  that  are  familiar,  have  tight  time  constraints,  and  involve  relatively 
low  stakes. 

•  At  the  EXPERT  stage,  learning  is  no  longer  significant.  The  degrees  of  automaticity, 
speed,  and  reliability  continue  to  rise,  albeit  more  slowly  than  before.  Learners  have 
become  accomplished  performers.  Recognition  and  timing  have  ceased  to  be  issues 
of  concern.  Experts  are  able  to  judge  and  apply  contextual  features  intuitively  and 
instinctively;  thus,  they  can  prune  an  option  tree  down  to  the  "first,  best"  option 
without  ever  explicitly  constructing  the  entire  tree.  In  most  cases,  a  satisfactory 
option  simply  comes  to  mind  at  the  same  time  the  decision  problem  is  recognized. 
Experts  are  extremely  efficient  in  uncertainty  reduction  and  can  "see"  entire  situation- 
goal-option  patterns  based  on  minimal  informational  cues.  Uncertainty  reduction  for 
the  expert  consists  essentially  of  probing  for  information  to  validate  a  perceived 
pattern,  and  of  mental  simulation  (or  rehearsal)  to  ascertain  feasibility  of  the  primary 
option,  which  is  frequently  the  only  option  under  consideration.  Decision  makers 
now  work  predominantly  in  the  RB  mode  with  some  work  being  accomplished  in  the 
ROC  mode  if  time  permits,  if  the  stakes  are  high,  and  if  the  problem  is  new  and 
unusual. 


A  more  concise  formulation  of  these  descriptions  is  presented  in  Table  5.  This  table,  in 
conjunction  with  Figure  12,  is  hereafter  referred  to  as  the  learning  model  for  decision-making 
skills  or  simply  as  the  learning  model. 
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Table  5.  Stages  in  Learning  Decision-Making  Skills 


...I  NOVICE 

ADVANCED  , 

EXPERT 

Knowledge 

General  knowledge, 

rudimentary, 

unstructured 

domain-specific 

knowledge 

Domain-specific 
facts,  concepts  & 
procedures, 
semistructured, 
loosely 

interconnected 

Full  scope  of 
domain-specific 
knowledge,  well 
structured,  highly 
interconnected 

Compiled,  highly 
interconnected 
domain  knowledge 

Recognition 

Slow,  unreliable 

Timely,  reliable 

Fast,  reliable 

Ahead  of  the  game 

Time  Line  Awareness 

None,  cannot  judge 
window 

Good,  usually 
correct  judgment  of 
window 

Very  good  reliable 
judgment  of 
window 

Not  an  issue 
anymore 

Use  of  Situational  or 
Context  Features 

Cannot  discriminate 
significant  from 
insignificant 

Can  discriminate 
and  apply,  albeit  not 
reliably 

Discriminates  and 
applies  reliably 

Extremely  acute 
discrimination 
based  on  minimal 

cues 

Prioritizing 

Uncertainty 

Cannot  prioritize  or 
estimate 

Begins  to  prioritize 

Prioritizes  correctly 
and  reliably 

Intuitive  recognition 

Uncertainty 

Reduction  Strategy 

Either  none  or  weak 
general  strategy 

Low-level,  partial 

domain-specific 

strategies 

Fully  developed? 

domain-specific 

strategies 

Intuitive  strategies 

Option  Set 

Generation 

Incomplete  sets,  no 
hierarchy 

■  Complete  sets, 
major  classes 

Complete  sets,  full 
hierarchical 
structures,  early 
pruning 

Only  as  much  as 
absolutely  necessary 

Mode 

ROC 

ROC 

ROC/RB 

RB 

Overall  Performance 

Slow,  analytical, 
unreliable, 
extremely  effortful, 
disjointed 

Slow,  analytical, 
reliable  with  simple 
problems,  effortful, 
serial 

Fast,  analytical, 
reliable  across 
spectrum,  low 
effort,  serial  with 
parallel  episodes 

Fast,  intuitive, 

I  reliable,  effortless, 
absorbed,  parallel 

3.3.4.  Training  Considerations 

Given  the  learning  model  described  above,  training  must  be  concerned  with 
accomplishing  the  transitions  from  one  stage  to  the  next  in  the  most  expedient,  efficient,  and 
effective  way  possible.  Given  the  usual  time  and  resource  constraints  under  which  training  must 
occur,  the  primaiy'  focus  of  a  formal  training  effort  should  be  on  the  first  two  transitions:  from 
novice  to  advanced  and  from  advanced  to  competent.  The  development  of  expertise  (i.e.,  the 
transition  from  competent  to  expert)  requires  massive  amounts  of  practice  in  the  actual  job 
environment  and  is  therefore  customarily  either  facilitated  by  intensive  and  formal  on-the-job 
training  (OJT)  or  simply  ignored.  Assuming  that  the  entry-level  student  falls  somewhere  in  the 
novice  band,  formal  training  should  concentrate  on  facilitating  the  transitions  from  novice  to 
advanced  (Transition  No.l)  and  from  advanced  to  competent  (Transition  No.  2).  What 
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foundation  of  domain-specific  knowledge  and  with  the  development  of  domain-specific 
procedural  skills.  Transition  No.  2,  on  the  other  hand,  is  concerned  with  refining,  integrating, 
and  accelerating  domain-specific  procedural  skills  and  with  the  development  of  the  performer's 
ability  to  apply  knowledge  of  underlying  domain  principles  to  practical  decision  problems.  The 
learning  requirements  that  must  be  addressed  by  the  two  transitions  are  therefore  fundamentally 
different. 

These  differences  in  learning  requirements  necessitate  differences  in  instructional 
treatments  or  methods.  The  first  transition  requires  methods  that  are  appropriate  for  remember- 
level  objectives  involving  facts,  concepts,  procedures,  and  principles  for  use;  find-level 
objectives  involving  concepts;  and  use-level  objectives  involving  procedures.  The  second 
transition  requires  methods  that  are  appropriate  for  use  and  find-level  objectives  involving 
procedures  and  principles. 

In  our  case,  the  choice  of  instructional  methods  is  constrained  by  the  a  priori  media 
choice  for  the  project  (i.e.,  by  the  requirement  to  provide  instructorless  training  on  a  desktop 
computer).  Computer-based  instructional  media  can  provide  instructional  treatment  in  two  basic 
forms.  The  first  is  commonly  referred  to  as  computer-assisted  instruction  (CAI);  the  second  is 
simulation.  CAI  implies  an  instructional  environment  where  a  carefully  structured  sequence  of 
lessons  and  lesson  segments  provides  training  in  the  building  blocks  of  some  terminal  skill.  CAI 
also  implies  that  explanations,  cues,  and  memory  aids  are  provided,  it  implies  extensive  practice 
and  feedback  in  component  skills.  Simulation,  on  the  other  hand,  implies  an  emphasis  on  part  or 
whole  task  practice  in  a  job-like  environment.  Explanations,  cues,  and  memory  aids  are  usually 
absent  in  simulation  per  se. 

Given  the  learning  requirements  for  the  first  transition,  we  can  broadly  assert  that  some 
form  of  CAI  would  be  the  instructional  method  of  choice.  With  equally  general  reasoning,  we 
can  assign  some  form  of  simulation  as  the  method  of  choice  for  the  second  transition. 

Simulation  would  not  work  for  the  first  transition  because  of  the  need  to  accommodate 
remember-level  objectives  and  because  of  the  need  to  train  procedures.  CAI  would  work  to 
some  extent  for  the  second  transition,  but  it  would  be  very  difficult  if  not  impossible  to  provide  a 
realistically  dynamic  decision-making  environment  with  this  method.  Thus,  the  following 
instructional  design  guidelines  have  been  formulated. 
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Instructional  Design  Guideline  No.  10 

To  facilitate  Transition  No.  1,  establish  a  foundation  of  domain-specific  knowledge  and 
procedural  skills  using  some  form  of  CAI.  Transition  No.  1  is  complete  when  the  learner 
can  reliably  solve  simple  decision-making  problems  without  aid. 


Instructional  Design  Guideline  No.  11 

&  To  facilitate  Transition  No.  2,  develop  integrated,  competent  levels  of  decision-making 
performance  using  some  form  of  realistic,  dynamic  simulation.  Transition  No.  2  is 
complete  when  the  learner  can  reliably  solve  complex  decision-making  problems  without 
aid. 


Instructional  Design  Guideline  No.  12 

I&  Do  not  attempt  to  transition  a  learner  firom  the  COMPETENT  stage  to  the  EXPERT  stage 
(Transition  No.  3)  by  means  of  formal  training.  Leave  the  development  of  expertise  to 
informal  OJT. 


The  efficiency  and  effectiveness  with  which  the  transitions  will  be  accomplished  depend 
heavily  on  the  type  and  quality  of  feedback  available.  The  role  of  feedback  in  learning  decision 
making  is  particularly  interesting  because  it  is  so  problematic.  Feedback  in  decision  making  can 
of  course  occm  only  after  the  decision  point;  however,  it  can  occur  before  the  start  of 
implementation  or  after  the  start  of  implementation.  Feedback  that  occurs  prior  to 
implementation  is  called  a  priori  feedback  and  feedback  that  occurs  as  a  result  of 
implementation  (sometime  after  implementation  has  started)  is  a  posteriori  feedback.  A  priori 
feedback  basically  answers  the  question  of  whether  the  decision  maker  has  made  a  decision  that 
is  both  optimal  and  logically  consistent  with  the  information  that  was  available  prior  to 
implementation.  A  posteriori  feedback  provides  information  on  whether  the  decision  was 
effective  ( i.e.,  whether  things  actually  worked  out  the  way  the  decision  maker  intended).  A 
priori  feedback  does  not  occur  in  the  real  world;  it  can  occur  only  in  a  "training  world." 
Therefore,  it  has  been  called  artificial  feedback  (Brecke  &  Young,  1990).  In  contrast,  a 
posteriori  feedback  has  been  called  natural  feedback  (ibid).  Natural  feedback  may  or  may  not 
actually  occur  in  the  real  world;  if  it  does  occur  it  is  often  confusing  and  difficult  to  attribute  to  a 
particular  prior  decision.  The  generation  of  natural  feedback  (i.e.,  feedback  that  has  all  the 


42 


salient  characteristics  of  a  posteriori  feedback  as  it  occurs  in  the  real  world)  is  usually  very 
difficult  in  a  simulation  environment 


Nickerson  and  Feehrer  (1975)  point  out  that  in  real-world  situations  decision  makers  are 
often  evaluated  on  the  basis  of  a  posteriori  feedback,  that  is,  on  the  basis  of  results  (e.g.,  "What 
really  counts  are  results").  This  is,  of  course,  not  very  logical  in  real-world  domains  where 
decision  outcomes  are  usually  subject  to  many  factors  that  are  beyond  the  decision  maker's 
control  —  perfectly  inane  decisions  may  turn  up  roses  and  really  clever  decisions  may  turn  into 
disasters.  Real-world  domains  are  basically  open-loop  systems,  and  in  open-loop  systems  the 
only  appropriate  manner  to  evaluate  a  decision  maker  is  on  the  basis  of  a  priori  feedback.  A 
posteriori  feedback  is  perfectly  fine  in  closed-loop  systems  where  all  factors  are  under  the 
decision  maker's  control,  but  such  systems  are  rarely  encountered  in  the  real  world  and  especially 
in  the  real  world  of  armed  conflict. 

Frequently,  people  must  learn  to  make  high-risk,  high-fi'equency  decisions  in  a  real-world 
domain  (i.e.,  an  open-loop  system)  without  the  benefit  of  instmction.  The  only  type  of  feedback 
available  to  them  is  a  posteriori  feedback  and,  when  such  feedback  arrives,  it  may  not  always  be 
clear  what  it  means  and/or  which  earlier  decision  it  belongs  to.  It  is  quite  possible  that  a  person 
under  this  set  of  circumstances  might  never  even  enter  a  learning  process,  much  less  complete  it 
successfully.  Of  course,  this  situation  can  be  improved  through  instruction  by  providing  a  priori 
feedback  and  by  providing  strategies  to  discover  and  properly  assess  a  posteriori  feedback,  or  by 
providing  both.  The  following  additional  guidelines  can  therefore  be  formulated: 


Instructional  Design  Guideline  No.  13 

I&  CAI  for  Transition  No.  1  must  include  practice  with  step-by-step  guidance  through  simple 
decision  problems  and  with  a  priori,  artificial  feedback. 


Instructional  Design  Guideline  No.  14 

&  Simulation  for  Transition  No.  2  must  include  realistic,  unaided  practice  with  both  artificial, 
a  priori  feedback  and  natural,  a  posteriori  feedback.  As  the  learner  gains  proficiency, 
artificial  feedback  should  be  withdrawn. 
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3.4.  What  Direct  Instructional  Design  Guidance  is  Available? 


3.4.1.  Introduction 

The  focus  now  turns  to  a  body  of  knowledge  that  concerns  itself  directly  with  the  design 
of  instruction.  Aagard  and  Braby's  (1976)  generic  instructional  strategy  for  decision  making  is 
the  most  direct  guidance  available;  therefore,  it  is  presented  first  and  discussed  in  detail. 

Guidance  available  from  instructional  design  theories  in  general  are  then  discussed  (Reigeluth, 
1983, 1987)  and  a  specific  set  of  theories  for  the  project  are  selected.  Finally,  we  look  at  two 
collections  of  "heuristics"  which  together  represent  knowledge  of  "what  works"  (Bennett,  1986; 
Montague,  1988)  in  instruction  and  extract  from  these  collections  heuristics  that  are  relevant  to 
training  decision-making  skills.  In  other  words,  this  section  examines  a  "mixed  bag"  of 
theoretically  derived  and/or  empirically  validated  guidance  that  is  available  to  instructional 
designers  who  endeavor  to  train  decision-making  skills. 

3.4.2.  Aagard  and  Braby’s  Instructional  Strategy  for  Decision-Making  Skills 

In  August  1975,  the  Interservice  Procedures  for  Instructional  Development,  also  known 
as  NAVEDTRA  106 A,  were  published.  This  six-volume  work  was  the  most  comprehensive  and 
detailed  attempt  undertaken  to  that  date  to  render  the  "art"  of  instructional  systems  development 
(ISD)  as  a  set  of  unambiguous  how-to  procedures  that  would  enable  subject  matter  experts  with 
very  little  training  to  generate  reliably  effective  and  efficient  instruction.  Contained  within  these 
volumes  are  learning  guidelines  and  algorithms  for  1 1  types  of  training  objectives  that  were 
developed  by  Aagard  and  Braby.  These  guidelines  were  also  published  seven  months  later  in  a 
separate,  less  voluminous  technical  report  (Aagard  &  Braby,  1976).  One  type  of  training 
objective  that  was  specifically  addressed  in  these  guidelines  was  decision  making.  Aagard  & 
Braby  identify  these  guidelines  as  "general  approaches"  that  are  "not  at  a  level  that  will 
accommodate  any  training  setting."  They  also  clearly  stated  that  they  "are  not  the  product  of 
empirical  research  but  are  the  product  of  rational  study"  and  that  they  had  "not  been  validated" 
and  that  "the  training  system  designer  should  adapt  these  solutions  into  more  sensitive  responses 
to  specific  requirements  of  the  learning  setting." 

Aagard  and  Braby  (1976)  first  present  definitions  of  the  1 1  categories  of  training 
objectives,  then  present  guidelines  for  instruction  in  both  prose  and  flowchart  form.  We 
reproduce  their  definition  of  the  "elemental  learning  task"  MAKING  DECISIONS  in  Table  6  and 
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provide  the  prose  guidelines  after  the  table.  The  definition  is  important  because  it  identifies 
(under  "EXAMPLES")  the  kinds  of  decision-making  skills  that  we  are  targeting  in  this  project. 

Table  6.  MAKING  DECISIONS  Described  as  a  Learning  Task  by  Aagard  and  Braby  (1976) 


NAMES  OF 
LEARNING 
TASKS 

lilliBII 

BEHAVIORAL  ATTRIBUTES 

EXAMPLES 

MAKING 

DECISIONS 

Choose 

Design 

Diagnose 

Develop 

Evaluate 

Forecast 

Formulate 

Organize 

Select 

1.  Choosing  a  course  of  action  when 
alternatives  are  unspecified  or 
unknown 

2.  A  successful  course  of  action  is 
not  readily  apparent 

3.  The  penalties  for  unsuccessful 
courses  of  action  are  readily 
apparent 

4.  The  relative  value  of  possible 
decisions  must  be  considered  — 
including  possible  trade-offs 

5.  Frequently  involves  forced 
decisions  made  in  a  short  period 
with  soft  information 

1 .  Choosing  frequencies  to  search 
in  an  Electronic  Counter 

Measures  (ECM)  search  plan 

2.  Choosing  torpedo  settings 
during  a  torpedo  attack 

3.  Assigning  weapons  based  on 
threat  evaluation 

4.  Choosing  tactics  in  combat  ~ 
wide  range  of  options. 

5.  Choosing  a  diagnostic  strategy 
in  dealing  with  a  malfunction  in 
a  complex  piece  of  equipment 

6.  Choosing  to  abort  or  commit 
oneself  to  land  upon  reaching 
the  critical  point  in  the  glide 
path 

Guidelines  for  Decision-Making  Training  (Aagard  &  Braby.  1976): 

MAKING  DECISIONS 

Decision  making  is  defined  here  as  the  application  of  a  specific  decision 
model,  thought  to  be  useful  in  diagnosing  equipment  malfunctions,  choosing 
tactics  in  Fleet  operations,  and  in  planning  where  several  alternatives  must  be 
considered,  each  with  an  unknown  probability  of  success.  The  decision  model 
combines  the  following  factors:  perception  of  the  problem,  identification  of 
alternative  solutions,  evaluation  of  these  alternatives,  and  selection  of  the  apparent 
best  solution.  Therefore,  the  guidelines  and  algorithms  presented  here  support 
learning  to  use  this  decision  model. 

The  decision-making  guidelines  presented  here  are  based  upon  the  most 
fashionable  practices  in  existing  decision-making  training  programs.  The 
following  guidelines  apply  to  decision-making  training. 

1 .  Ensure  that  the  student  acquires  the  knowledge  required  to: 
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a.  identify  the  problem 

b.  generate  reasonable  solutions 

c.  evaluate  these  solutions 

2.  Decrease  student  anxiety  to  a  low  level,  particularly  in  the  early  stages  of 
learning,  where  student  anxiety  is  high  and  where  complex  decisions  are  to  be 
made. 

3.  Give  the  student  examples  of  these  two  types  of  actions  which  are  to  be 
avoided  when  making  a  decision: 

a.  The  tendency  to  make  a  "favorite"  decision  or  to  use  a  "favorite" 
solution  regardless  of  the  real  nature  of  the  problem 

b.  The  tendency  to  generalize  problems  or  view  several  types  of  problems 
as  if  they  were  all  the  same  when,  in  fact,  they  are  quite  different 

Give  examples  of  these  undesirable  responses  in  decision  making. 

4.  Teach  a  decision-making  strategy;  the  following  strategy  is  suggested: 

a.  Upon  becoming  aware  of  the  problem,  define  it 

b.  Be  alert  for  the  availability  of  relevant  information  and  collect  such 
data 

c.  Develop  alternative  solutions 

(1)  State  alternative  solutions 

(2)  Combine  alternative  solutions 

(3)  Compare  alternative  solutions 

d.  Evaluate  alternative  solutions 

(1)  List  the  probable  consequences  of  each  alternative  solution 

(2)  Rank  each  alternative  solution  according  to  desirability  of 
consequences 

e.  Choose  course  of  action  based  on  a  desired  solution 

f.  Execute  the  chosen  course  of  action 

5.  Vary  the  setting  of  the  significant  cues  of  the  decision-making  learning  task. 
Provide  both  basic  and  advanced  problems  to  be  solved  with  a  wide  range  of 
problem  difficulty  at  each  level  of  training  for  the  operational  tasks. 

6.  Ensure  the  overlearning  of  decision-making  skills  in  later  stages  of  training  if 
the  student  will  be  required  to  perform  under  stress  in  the  real  world. 
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7.  Present  the  student  with  a  realistic  data  load  (i.e.,  realistic  amount  of 
significant  data)  plus  operational  distractors  in  real  time  towards  the  end  of 
training. 

8.  Provide  the  student  with  access  to  potentially  relevant  data  during  practice. 
In  the  final  stages  of  training,  the  data  available  to  him  should  be  limited  to 
that  expected  in  the  real-world  situations  in  which  he  will  be  working. 

9.  Feedback:  Provide  the  student  with  answers  to  the  five  following  questions 
after  his  decisions  in  practice  problems.  These  answers  serve  as  Knowledge 
of  Results  (KR). 

a.  Predictability?  (Were  problems  mistakenly  viewed  as  if  they  were  all 
the  same  in  reaching  solutions?) 

b.  Persistence?  (Was  use  made  of  a  "favorite"  solution  when  it  was 
inappropriate?) 

c.  Time  liness?  (Was  this  the  appropriate  time  to  execute  this  particular 
decision?) 

d.  Completeness?  (Was  all  of  the  available  information  considered?) 

e.  Consistency?  (Was  the  solution  compatible  with  and  relevant  to  the 
available  information?) 

Give  the  KR  with  respect  to  the  five  criteria  each  time  the  student  makes  a 
decision  and,  if  possible,  provide  the  simulated  consequences  of  the  decision  as 
compared  to  alternative  solutions. 


This  instructional  strategy  is  remarkably  comprehensive  in  terms  of  the  variety  of 
concerns  it  addresses.  Its  recommendations  have  a  great  deal  of  face  validity,  particularly  in 
view  of  the  fact  that  there  was  not  much  of  a  research  base  from  which  to  derive  that  strategy  and 
that  its  source  was  apparently  nothing  more  than  "most  fashionable  practice."  It  is  directly 
applicable  only  to  decision-making  tasks  where  option  generation/evaluation  is  the  main 
problem.  Nevertheless,  this  strategy  is  a  good  starting  point  and  it  represents  the  one  and  only 
explicitly  formulated  instructional  strategy  for  training  decision-making  skills  that  can  be  found 
in  the  literature.  Aside  from  its  relatively  narrow  range  of  applicability,  there  are  a  number  of 
other  points  in  this  strategy  that  deserve  to  be  highlighted  either  negatively  or  positively.  On  the 
positive  side  are  the  emphasis  on  ensuring  the  acquisition  of  prerequisite  domain  knowledge,  the 
emphasis  on  teaching  the  meta-knowledge  of  "how  to  make  a  decision"  and  "the  mistakes  to 
avoid,"  and  the  remarkable  detail  on  ensuring  a  realistic  data  load  and  appropriate  distractors 
during  later  stages  of  learning. 
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There  are  also  several  negative  side  aspects.  First  is  the  somewhat  inconsistent  advice  on 
how  to  deal  with  the  stress  issue.  Decreasing  the  student's  anxiety  through  some  instructional 
manipulation  may  not  be  appropriate,  particularly  when  the  real  setting  will  be  a  high-stress 
environment.  Rather,  anxiety-promoting  conditions  may  be  more  useful  in  training  the  student  in 
coping  strategies.  The  recommendation  to  ensure  "overlearning"  is  one  possible  coping  strategy, 
but  it  is  not  the  only  one  and  it  might  be  very  costly  to  implement.  The  treatment  of  human 
foibles  and  inadequacies  in  decision  making  is  really  limited  to  a  guard  against  inappropriate  use 
of  the  availability  and  representativeness  heuristics  (Kahnemann,  Slovic,  &  Tversky,  1982). 
However,  as  Section  3.2  indicates,  there  is  much  more  that  can  be  —  and  probably  should  be  — 
added  on  that  score.  Then  there  is  a  recommendation  to  provide  access  to  potentially  relevant 
data  during  practice  and  to  limit  that  access  to  real-world  levels  during  later  training  stages.  This 
is  good  advice,  but  it  also  indicates  a  lack  of  recognition  for  the  pivotal  role  that  information 
acquisition  really  plays  in  decision-making  tasks.  It  is  paramount  that  students  learn  what 
information  sources  are  available  in  their  real  environment,  how  reliable  fcose  sources  are,  how 
they  can  access  them,  and  what  the  cost  of  information  acquisition  is  in  time  and/or  other 
resources.  Last  but  not  least,  the  recommendations  for  feedback,  although  interesting,  appear  to 
be  very  difficult  to  implement  (especially  predictability,  persistence,  and  completeness). 
Providing  simulated  decision  consequences  is  especially  difficult  and,  as  discussed  in  the 
previous  section,  might  lead  to  misinterpretations  when  the  consequences  are  also  influenced  by 
factors  other  than  the  student's  decision. 

In  short,  Aagard  and  Braby's  (1976)  strategy,  although  the  best  published  strategy,  is 
limited.  It  is  perhaps  too  general  to  produce  reliable  training  results,  however,  it  provides  a  good 
idea  of  the  sort  of  thing  that  instructional  designers  ought  to  have  available.  It  is  a  fairly 
comprehensive  collection  of  instructional  prescriptions,  but  one  that  is  based  more  on  spurious 
evidence  of  what  works  here  and  there  rather  than  on  a  cohesive  and  internally  consistent  theory 
of  what  should  work  everywhere.  Such  theories  are  considered  in  the  following  paragraphs. 

3.4.3.  Instructional  Design  Theories 

The  primary  puipose  of  the  science  of  instructional  design  (ID)  is  to  prescribe  optimal 
methods  of  instruction  (Reigeluth,  1983).  Of  interest  in  this  study  are  those  prescriptive  ID 
theories  that  can  provide  guidance  for  the  design  of  optimal  training  in  decision-making  skills. 
Fortunately,  all  serious  candidate  theories  that  might  help  us  in  this  regard  have  been 
conveniently  collected,  described,  and  discussed  in  two  volumes  edited  by  Reigeluth  (1983, 
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1987).  These  two  volumes  represent  an  outstanding  integrative  effort  that  provides  a 
comprehensive  picture  of  the  current  state  of  the  art  in  ID  theory. 

All  these  theories,  whether  the  foxmdational  Gagne-Briggs  Theory  of  Instruction, 
Cropper's  Behavioral  Approach  to  Instructional  Prescription,  Landa's  Algo-Heuristic  Theory  of 
Instruction,  Scandura's  Structural  Learning  Theory,  Collins  and  Steven’s  Cognitive  Theory  of 
Inquiry  Teaching,  Merrill's  Component  Display  Theory,  Reigeluth's  Elaboration  Theory  of 
Instruction,  or  Keller's  Motivational  Design  of  Instruction,  proceed  from  the  common  assumption 
that  different  sets  of  desired  instructional  outcomes  and  given  instructional  conditions  require 
different  instructional  methods.  They  also  all  offer  a  "language"  of  concepts  and  elements  of 
instructional  design  as  well  as  prescriptions  for  the  optimal  arrangement  of  these  elements. 

There  are  many  overlaps  between  these  theories,  and  Reigeluth  has  incisively  commented  on  the 
similarities  and  differences  of  language  and  prescription  offered  by  these  theories. 

The  fundamental  question  for  the  instructional  designer  who  faces  this  banquet  of 
available  theories  is  whether  to  select  one,  synthesize  them  into  a  new  one,  or  pick  and  choose 
from  each.  The  last  alternative  is  difficult  to  do  systematically  unless  one  uses  some  set  of 
principles  —  some  sort  of  meta-theory.  However,  such  a  meta-theory  does  not  exist.  The 
synthesis  alternative  is  desirable  but  goes  far  beyond  the  scope  of  this  project.  The  only  feasible 
alternative  is  to  select  a  theory  and  to  use  it  as  the  "bedrock"  foimdation  for  the  design  of  an 
instructional  method  that  uniquely  fits  the  desired  instructional  outcomes  and  the  given 
instructional  conditions  or  constraints.  That  does  not  mean,  in  our  opinion,  that  one  cannot 
supplement  such  an  approach  with  principles  originating  from  other  theories,  especially  if  the 
selected  theory  has  missing  and/or  deficient  aspects.  This  is  different  from  a  "pick  and  choose" 
approach  because  it  does  not  rely  on  one  theory  as  the  predominant  source  of  guidance;  in  fact,  it 
might  more  appropriately  be  taken  for  a  synthesis,  albeit  one  that  starts  with  a  bias  towards  one 
particular  theory.  This  approach,  which  could  be  called  the  "select  and  supplement"  approach,  is 
actually  used  in  this  project. 

Adopting  this  approach  requires  the  establishment  of  criteria  for  selecting  principles. 
Reigeluth  (1983)  suggests  four  criteria  for  evaluating  instructional  design  theories:  (1) 
comprehensiveness,  (2)  optimality  or  usefulness,  (3)  breadth  of  application,  and  (4)  parsimony. 
The  comprehensiveness  criterion  is  particularly  useful,  because  it  evaluates  whether  the  theory 
accoimts  for  all  "classes  of  methods”:  organizational  (both  macro  and  micro),  delivery,  and 
management.  Optimality  or  usefulness  is  a  rather  "soft"  criterion  because  it  would  seem  difficult 
to  determine  which  theory  is  optimal  for  a  given  purpose  imless  one  actually  produces  test 
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samples  of  instruction  and  tries  them  out.  Breadth  of  application  refers  to  the  number  of 
conditions  "under  which  the  model  is  optimal"  and  to  the  number  of  desired  outcomes  "for  which 
it  is  optimal."  This  criterion  falls  into  the  same  trap  of  requiring  an  empirical  test,  but  it  can  be 
used  if  "for  which  it  is  optimal"  is  replaced  with  "for  which  it  is  designed."  Parsimony,  finally,  is 
always  an  admirable  criterion,  especially  when  one  does  not  have  much  money. 

We  have  made  our  selection  on  the  basis  of  comprehensiveness,  parsimony,  and  logical 
clarity  and  have  decided  to  use  Reigeluth's  Elaboration  Theory  for  the  macro  aspects  of 
instructional  strategy  and  Merrill's  Component  Display  Theory  for  the  micro  aspects.  These  two 
theorists  have  in  fact  collaborated  to  produce  a  complementary  theoretical  framework  that 
provides,  in  combination,  the  most  comprehensive  and  the  most  clearly  and  simply  articulated 
guidance  available.  However,  neither  of  these  two  complementary  theories  deals  with  the 
motivational  aspects  of  instruction  in  any  more  than  cursory  form.  We  therefore  supplement 
these  two  theories  with  Keller's  ideas  for  the  motivational  design  for  instraction.  With  this 
decision,  we  have  available  to  us  a  comprehensive  set  of  prescriptive  principles  that  can  provide 
guidance  in  the  design  of  the  macro  and  micro  aspects  of  an  instructional  strategy  tailored  to  the 
requirements  of  training  decision-making  skills. 

Contrary  to  the  Aagard  and  Braby  guidelines,  the  Reigeluth-plus-Merrill-plus-Keller 
(RMK)  guidelines  are  focused  exclusively  on  the  issues  of  HOW  to  teach  rather  than  on  WHAT 
to  teach,  and  they  do  provide  more  detail,  more  breadth,  and  more  discipline  in  this  respect. 

What  the  RMK  guidelines  are  and  how  they  can  be  applied  to  the  design  of  an  instructional 
strategy  for  decision-making  skills  will  be  discussed  in  the  next  section  of  this  paper.  Before  we 
turn  to  this  next  section,  however,  we  shall  examine  one  last  set  of  sources  that  yield  additional 
guidelines  for  our  purpose. 

3.4.4.  Instructional  Engineering  Heuristics 

In  1986,  the  U.S.  Department  of  Education  published  a  booklet  entitled:  "What  Works: 
Research  about  Teaching  and  Learning"  (Bennett,  1986).  This  precipitated  the  development  of  a 
very  similar  document  entitled:  "What  Works:  Summary  of  Research  Findings  with 
Implications  for  Navy  Instruction  and  Learning"  (Montague,  1988).  The  first  document 
addresses  research  findings  that  apply  to  the  teaching  and  learning  of  children  (i.e.,  to  primary 
and  secondary  education).  The  second  document  addresses  research  findings  that  apply  to  the 
teaching  and  learning  of  adults,  which  makes  this  latter  document  particularly  relevant  for  our 
purposes.  Both  documents  present  collections  of  what  Montague  (1987)  has  called  "instructional 
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engineering  heuristics"  or  guidelines  for  the  arrangement  of  instruction.  Each  guideline  is  based 
on  reliable  research  findings  that  have  been  confirmed  repeatedly  in  different  circumstances. 
Montague's  "What  Works"  collection,  like  Bennett's  collection,  is  subdivided  into  heuristics  that 
work  for  training  executives,  for  instructors,  and  for  training  specialists.  Nineteen  out  of  48  of 
these  heuristics  are  directly  applicable  to  our  project.  Most  of  these  heuristics  confirm  guidance 
already  obtained  from  other  sources.  However,  there  is  one  that  is  viewed  as  particularly  useful: 
the  use  of  mental  models  in  training  and  performance.  We  present  this  entire  heuristic  in  Table  7 
to  introduce  its  technical  content  and  to  show  how  these  heuristics  are  presented.  We  also 
present  the  research  findings  for  each  of  the  19  heuristics  and  describe  how  we  intend  to  apply 
them  to  the  design  of  the  instructional  strategy  for  decision  making  (Table  8). 


Table  7.  Instructional  Design  Heuristic  after  Montague  (1988) 


Promote  Development  of  Mental  Models 

Finding:  When  students  are  asked  to  act  in  accordance  with  a  prescribed  '‘model"  of  performance,  they  develop  conceptual 

understanding  that  guides  competent  performance  more  effectively. 

Comments:  Learning  involves  the  development  of  qualitative  conceptual  structures  that  are  called' "mental  models."  A  person  makes 

use  of  an  internal  model  of  the  world  to  understand,  explain,  and  predict  things  about  the  world.  If  people  carry  a  small- 
scale  model  of  external  reality  in  their  heads,  they  are  able  to  try  out  various  alternatives,  decide  which  of  them  is  best, 
react  to  future  situations  before  they  occur,  and  utilize  knowledge  of  past  events  in  dealing  with  the  present  and  future. 
Models  allow  people  to  generate  descriptions  of  system  purpose  and  form  or  explain  system  functioning  and  observed 
slates,  and  to  make  predictions  of  future  states.  These  models  provide  a  means  for  organizing  and  reorganizing  memory 
and  deciding  on  actions. 

Mental  models  evolve  naturally  through  the  interaction  of  the  learner  and  particular  environments.  If  this  is  so,  we  can 
devise  methods  to  promote  their  development.  One  way  is  representing  the  functionality  of  the  work  environment  and 
the  devices/equipment  in  it.  In  addition,  providing  external  guidance  or  directions  (i.e.,  telling  what  to  do  and  how  to  do 
it)  allows  the  buildup  of  experience  coupled  with  important  cognitive  information  that,  once  internalized,  will  guide 
performance.  An  accurate  mental  model  develops  from  the  way  events  flow  on-the-job,  how  devices  function  and  can 
malfunction,  and  serves  as  the  scheme  to  guide  personal  action  when  new  problems  are  encountered.  Having  students 
describe  in  detail  the  steps  they're  using  while  performing  identifies  errors,  develops  competence  faster,  and  transfers 
readily  to  the  work  environment. 

As  an  example,  consider  the  task  of  training  students  to  solve  problems  in  electric  circuits,  thermodynamics,  or 
mechanics.  By  guiding  students  through  the  steps,  explaining  why  they're  taken,  then  having  students  describe  the 
factors  and  their  interactions  as  they  solve  subsequent  problems,  they  learn  rapidly  and  accurately.  Instructors  can  check 
the  accuracy  of  students’  initial  representation  and  provide  feedback.  It  focuses  students'  attention  on  the  need  for  careful 
representation  of  all  facets  of  the  problem  and  provides  the  basis  for  correct  solutions.  Thus,  by  concentrating  on 
accurate  initial  description  of  the  problem,  students  learn  to  internalize  the  procedures  as  part  of  their  mental  model, 
w  hich  they  use  habitually  in  approaching  later  problems. 

References:  Anderson,  R.  C  (1977).  The  notion  of  schemata  and  the  educational  enterprise:  General  discussion  of  the  conference.  In 

R.  C.  Anderson,  R.  J.  Spiro  &  W.  E.  Montague  (Eds.),  Schooling  and  the  acquisition  of  knowledge  (415-431). 
Hillsdale,  NJ:  Erlbaum  Associates  415-431. 

Centner,  D.,  &  Stevens,  A.  L.  (1983).  Mental  models.  Hillsdale,  NJ:  Erlbaum  Associates. 

Heller,  J.  1.,  &  Reif,  F.  (1984).  Prescribing  effective  human  problem-solving  processes:  Problem  description  in  physics. 
Cognition  and  Instruction,  1(2),  177-216. 

Kieras,  D.  E.  (in  press).  What  mental  model  should  be  taught:  Choosing  instructional  content  for  complex  engineered 
systems.  In  J.  Psotka,  D.  Massey,  and  S.  Mutter  (Eds.),  Intelligent  tutoring  systems:  Lessons  learned.  Hillsdale, 
NJ:  Erlbaum  Associates. 
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Table  8.  Research  Findings  and  Comments  from  "What  Works"  (Montague,  1988) 


No. 

Research  Finding 

Comments 

1 

Students  learri  the  same  content  as  well  or  better  from 
computer-based  instruction  as  in  a  regular  classroom  situation 
and  complete  the  lessons  faster;  course  materials  can  be  widely 
distributed  and  given  at  any  time. 

This  simply  supports  the  idea  for  a  desktop 
computer-based  trainer.  The  cost 
effectiveness  of  the  trainer  will  be  largely 
dependent  on  how  widely  distributed  it  is. 

2 

Students  can  learn  as  well  from  structured  instructional 
material  and  self-study  as  from  conventional  classroom 
procedures. 

This  confirms  the  above.  The  training 
system  will  incorporate  a  structured  course 
for  self-study  (i.e.,  a  CAI  component). 

3 

When  instruction  gets  the  student's  attention,  is  perceived  as 
relevant  and  as  having  attainable  goals,  and  provides  frequent 
testing  and  explanatory  feedback,  students  work  hard,  achieve 
well,  and  enjoy  learning. 

These  are  the  essential  tenets  of  Keller's 
Motivational  Design  for  Instraction  (the  K  in 
RMK). 

■ 

Students  learn  best  when  instruction  is  adapted  to  their  existing 
knowledge  and  background. 

All  students  will  be  personnel.  The  subject 
matter  will  be  an  artificial  or  fictional 
domain. 

5 

Students  who  spend  as  much  time  as  possible  actively  engaged 
in  learning  leam  more  than  students  who  do  not. 

Instruction  in  both  the  CAI  component  and 
the  Simulation  component  will  be 
continuously  interactive. 

6 

Students  perform  best  when  their  instructors  inspire  them  to 
take  an  active  role  in  their  learning. 

The  instmctional  strategy  will  incorporate 
options  for  various  levels  of  learner  control. 

■ 

Learning  improves  when  students  know  how  to  set  and  achieve 
their  own  goals. 

The  instmctional  strategy  will  incorporate 
options  for  various  levels  of  learner  control. 

8 

Diagrams,  graphs,  photographs,  and  illustrations  can  improve 
student  learning. 

Such  devices  will  be  used  where  appropriate. 
They  are  standard  ingredients  of  well- 
designed  instmctional  materials. 

9 

Enhancement  of  text  in  books  or  manuals  through  orientation, 
summaries,  examples,  and  diagrams  can  aid  student 
comprehension  and  learning. 

These  kinds  of  enhancements  are  already  part 
of  the  RMK  guidelines  and  they  are 
customarily  employed  in  CBT. 

10 

When  students  act  in  accordance  with  a  prescribed  "model"  of 
performance,  they  develop  conceptual  understanding  that 
guides  competent  performance  more  effectively. 

Instmction  will  include  mental  models  for 
decision  making  and  for  systems  and 
processes  describing  and  explaining  the 
technical  domain. 

11 

The  ways  students  study  influence  what  and  how  much  they 
leam.  Students  can  leam  effective  study  strategies. 

This  suggests  the  inclusion  of  objectives  and 
content  dealing  with  study  skills  for  decision 
making  with  the  device  we  are  building. 

12 

Providing  students  with  representative  good  examples  and 
contrasting  them  with  bad  examples  teaches  them  the  desired 
knowledge  and  skills. 

This  is  part  of  Merrill's  Component  Display 
theory  (the  M  in  the  RMK  guidelines). 

13 

Practicing  lesson-related  tasks  promotes  learning  new  skills. 

Levels  of  practice  will  be  one  of  the  major 
strategy  variables  that  can  be  controlled. 

14 

Students  leam  and  retain  knowledge  and  skills  best  when  the 
learning  environment  incorporates  the  critical,  functional 
features  of  the  working  environment. 

The  critical,  functional  features  for  a  working 
&  decision-making  environment  are  the 
eliciting  stimuli,  the  available  information 
resources  for  the  reduction  of  uncertainty,  the 
response  tools,  and  the  feedback  stimuli. 

They  will  be  replicated  in  the  training 
environment  with  functional  rather  than 
physical  fidelity. 

52 


Table  8.  Research  Findings  and  Comments  from  "What  Works"  (Montague,  1988)  (Continued) 


No. 

Research  Finding 

Comments 

15 

A  simulator’s  effectiveness  is  a  fimction  of  the  instructional 
methods  incorporated  into  it  to  support  student  learning; 
design  decisions,  therefore,  must  be  related  to  the  cognitive 
processes  required  to  learn  the  task. 

This  is  precisely  the  reason  we  are 
developing  this  paper  and  the  reason  we  will 
develop  our  design  for  the  simulation 
module  based  on  guidelines  developed  here. 

16 

Effective  simulation  provides  systematic  practice  and  feedback 
about  errors,  depicts  how  a  device  or  system  works,  and  may 
violate  physical  and  temporal  fidelity. 

Systematic  practice  and  feedback  are  part  of 
the  RMK  guidelines.  The  ’’system"  will  be 
artificial  (i.e.,  it  will  violate  physical 
fidelity). 

17 

Frequent,  systematic  testing  and  assessing  of  student  progress 
informs  students  about  their  learning  and  informs  instructors 
and  managers  about  strengths  and  weaknesses  in  student 
learning  and  the  instruction. 

Frequent,  systematic  testing  will  be  part  of 
the  instructional  strategy. 

18 

Students  who  receive  constructive  feedback  about  the  accuracy 
and  adequacy  of  their  performance  become  more  interested  in 
the  class  and  learn  more. 

Constructive  feedback  will  be  a  motivational 
aspect  of  the  instructional  strategy. 

19 

Maintaining  critical  skills  requires  systematically  planned  and 
monitored  on-the-job  rehearsal  and  testing. 

The  training  system  we  are  building  makes 
skill  maintenance  in  decision  making  a 
practical  possibility.  It  will  be  up  to  the 
users  to  avail  themselves  of  it.  This  finding 
provides  support  for  leaving  the  development 
of  expertise  to  OJT. 

3.4.5  Training  Considerations 

This  section  deals  with  training  considerations  and  examines  three  sources:  an  off-the- 
shelf  strategy,  instructional  design  theories,  and  two  collections  of  instructional  engineering 
heuristics  that  work  in  a  wide  variety  of  circumstances.  These  three  sources  offer  a  plethora  of 
often  redundant  instructional  recommendations  that  could  easily  be  consolidated  into  a  smaller, 
more  manageable  set  of  nonredundant  instructional  design  guidelines.  There  is  one 
encompassing  and  general  guideline  that  is  based  on  the  choice  of  instructional  design  theories 
made  above: 


Instructional  Design  Guideline  No.  15 

I&  Design  the  organizational  strategy  for  training  decision-making  skills  using  Reigeluth's 
elaboration  strategy  as  guidance  for  the  macro  strategy  and  Merrill's  CDT  for  the  micro 
strategy.  Complement  either  with  elements  based  on  Keller's  theory  on  motivational  design 
of  instruction. 
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This  guideline  implies,  of  course,  entire  sets  of  more  specific  instructional  design 
guidelines.  These  lower-level  guidelines  are  not  reproduced  here  but  are  referred  to  in  Section 
4.0,  which  focuses  on  the  actual  application  of  all  guidelines  developed  in  Section  3.0  (the 
present  section).  The  four  guidelines  below  are  more  specific  than  the  guideline  above,  and  they 
relate  primarily  to  the  micro  aspects  of  instruction.  They  are,  in  all  cases,  guidelines  that  have 
been  consolidated  by  summarizing  several  guidelines  from  either  Aagard  and  Braby  or 
Montague,  or  from  both.  The  source  of  each  guideline  is  indicated  in  parentheses.  When  a 
guideline  corresponds  to  a  more  specific  guideline  from  Reigeluth's,  Merrill's,  or  Keller's 
theories,  the  parentheses  include  an  R,  M,  or  K,  as  the  case  may  be. 


Instructional  Design  Guideline  No.  16 

&  Training  content  should  include  relevant  technical  domain  knowledge  and  meta-knowledge 
on  decision  making.  Where  appropriate,  the  content  of  either  type  should  be  cast  in  the  form 
of  mental  models  of  prescribed  performance.  Where  appropriate,  such  content  should  also 
include  contrasting  good  and  bad  examples.  Training  content  may  inelude  study  skills. 

(A&B:  1., 3. ,4.;  Mont.:  10, 11;M)  _ _ 


Instructional  Design  Guideline  No.  17 

&  Practice  should  be  realistic  and  frequent,  cover  a  wide  range  of  difficulty  and  significant 
cues,  and  provide  constructive  feedback  that  is  directly  related  to  training  objectives. 

(A&B:  5.  through  9.;  Mont:  13  through  18;  M) _ 


Instructional  Design  Guideline  No.  18 

&  Enhance  student  motivation  by  getting  the  student's  attention,  establishing  relevance,  setting 
attainable  goals,  providing  for  ample  practice  and  feedback,  and  permitting  learner  control 
whenever  possible. 

(Mont.:  3  through  7,  K)  _ 


Instructional  Design  Guideline  No.  19 

&  Enhance  the  presentation  of  instructional  content  with  orientation  aids,  summaries,  and 
syntheses,  and  with  clear,  appropriate  illustrations  of  various  types. 

(Mont:  8,9:  M,  K)  _ 
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Instructional  Design  Guideline  No.  20 

&  During  early  stages  of  learning,  student  anxiety  should  be  kept  low.  Later  stages  of  learning 
should  begin  to  introduce  realistic  levels  of  stress. 

(A&B:  2.)  _ 
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IV.  APPLICATION 


4.1.  Introduction 


This  section  attempts  to  apply  the  instructional  design  guidelines  developed  in  the 
preceding  sections  to  the  logistics  training  problem.  Before  doing  so,  it  is  useful  to  articulate  the 
problem  once  again:  We  are  searching  for  an  optimal  method  to  train  decision-making  skills 
under  certain  conditions  and  constraints.  The  conditions  and  restraints  are  given  by  the  target 
population  of  learners,  by  the  subject  matter  domain,  by  the  training  goal,  and  by  the  teaching 
agent  or  delivery  medium: 

•  The  learners  are  adult  military  or  civilian  personnel  who  will-occupy  positions  with 
decision-making  functions  in  Command  and  Control  (C^)  agencies  during  times  of 
heightened  tension,  crisis,  or  war.  They  may  or  may  not  occupy  the  same  positions 
during  peacetime. 

•  The  subject  matter  domain  in  its  most  general  form  can  be  described  as  C^  and  in 
more  specific  form  as  Logistics  Command  and  Control  (LC^),  where  decision  making 
is  concerned  with  the  assignment  of  resources  to  operational  demands  on  the  basis  of 
multiple,  often  interrelated  factors. 

•  The  training  goal  is  to  achieve  a  level  of  learner  decision-making  skill  that  will 
enable  learners  to  function  at  high  levels  of  effectiveness  immediately  upon 
occupying  &  positions  under  operational  conditions.  This  level  of  skill  is  identified 
as  the  COMPETENT  stage  in  the  learning  model  presented  in  Section  3.3. 

•  The  teaching  agent  or  delivery  medium  is  an  IBM  PC-compatible,  386-based 
desktop  computer  with  a  color  monitor  available  under  the  DESKTOP  III 
procurement. 

The  strategy  for  the  pursuit  of  the  optimal  training  method  is  to  first  reduce  the  search 
space  by  theoretical  analysis,  to  formulate  a  strawman  methodology  based  on  the  results  of  the 
analysis,  and  finally  to  refine  the  strawman  methodology  through  empirical  trials.  We  have 
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presented  the  theoretical  analysis  in  the  preceding  sections  and  are  now  ready  to  formulate  a 
strawman  instructional  method. 

To  define  an  instructional  method,  strawman  or  otherwise,  one  must  specify 
organizational,  delivery,  and  management  strategies  (Reigeluth,  1983).  In  our  case,  the  delivery 
strategy  is  essentially  predetermined  (desktop  computer-based  training).  That  leaves 
organizational  and  management  strategies  as  open  issues.  In  this  section,  we  define  a  strawman 
organizational  strategy  and  deal  with  the  issue  of  management  strategy  in  the  context  of  needs 
for  empirical  research. 

The  remainder  of  this  section  consists  of  four  parts.  The  first  part  deals  with  the  training 
target.  It  begins  with  a  discussion  of  what  types  of  decision-making  tasks  warrant  the 
expenditure  of  training  resources  and  ends  with  the  specification  of  the  terminal,  training 
objective  for  the  system  to  be  built.  The  second  part  specifies  the  content  of  the  training.  The 
third  part  outlines  the  strawman  organizational  strategy  for  training  decision-making  skills  on 
both  a  macro  and  micro  level.  The  fourth  and  last  part  provides  a  brief  overview  of  some  major 
research  issues  that  will  have  to  be  investigated  along  the  road  to  the  optimal  instructional 
strategy  for  decision  training. 

4.2.  Training  Target 

Training,  especially  in  times  of  limited  funding,  should  concentrate  on  those  types  of 
decision-making  tasks  that  represent  the  most  "lucrative  targets"  for  training.  Clearly,  decision¬ 
making  tasks  that  are  firequently  performed  by  many  people,  that  are  difficult  to  learn  and 
perform,  that  involve  high  stakes,  and  for  which  there  is  currently  no  adequate  training  are 
lucrative  targets.  They  are  likely  to  show  a  greater  return  for  any  investment  in  training  than 
tasks  with  the  opposite  characteristics.  This  means,  in  practical  terms,  that  the  decision-making 
tasks  performed  by  a  single  Commander-in-Chief  (CINC)  are  probably  less  appropriate  targets 
for  training  than  the  many  decisions  made  every  hour  by  the  many  captains,  majors,  and 
lieutenant  colonels  at  lower  levels  of  the  military  hierarchy.  There  may  be  much  more  at 
stake  and  it  may  be  much  more  difficult  to  make  CINC  decisions  and  to  learn  to  make  them,  but 
one  can  safely  assume  that  one  does  not  become  a  CINC  unless  one  demonstrates  high  levels  of 
skill  in  decision  making  along  the  way. 

Decision-making  tasks  that  score  high  on  task  frequency,  number  of  performers, 
difficulty  to  learn  and  perform,  mission  criticality,  and  inadequacy  of  current  training  are  easily 
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found  in  agencies  (Brecke  et  al.,  1988;  Schwaninger  et  al.,  1991).  Within  such  agencies  one 
generally  finds  these  tasks  at  the  mid-management  level.  This  was  confirmed  during  site  visits  at 
Headquarters  Air  Force  Logistics  Command  (AFLC),  Air  Logistics  Center  (ALC)  Ogden,  and 
ALC  Warner  Robins.  Decision-making  tasks  with  the  desired  characteristics  were  easily  foxmd 
at  the  ALCs  in  positions  with  three-  and  four-letter  office  symbols.  They  appeared  to  be  less 
frequent  at  higher  levels  within  the  ALCs,  and  they  were  not  found  at  all  at  Headquarters  AFLC. 

On  the  surface,  there  seems  to  be  an  endless  variety  of  decision-making  tasks,  but  this 
variety  likely  can  be  reduced  to  a  manageable  number  of  categories.  One  notable  attempt  to 
categorize  tactical  decision-making  tasks  was  the  ACADIA  taxonomy  proposed  by  Sidorsky  and 
Houseman  (1966)  in  which  decisions  were  classified  as  acceptance,  change,  anticipation, 
designation,  implementation,  and  adaptation  decisions.  Although  the  ACADIA  scheme  is  less 
than  convincing,  there  is  no  reason  tactical  decision  problems  caimot  be  reduced  to  a  few 
"classical"  types.  We  believe  we  have  identified  one  such  type  with  certmnty  (which  is  of  course 
a  long  way  from  an  exhaustive  taxonomy).  We  have  called  this  type  of  tactical  problem  the 
"Demand  and  Supply"  (DS)  problem. 

The  DS  problem  arises  when  operational  units  request  resources.  The  decisions  that  must 
be  made  either  deal  with  ways  and  means  to  satisfy  a  given  request  or,  if  requests  are  competing 
for  limited  resources,  with  distribution  of  resources  over  a  number  of  requests.  An  example  of 
the  latter  is  a  situation  in  a  theater  where  several  users  (wings,  squadrons)  request  more 
ammunition  of  a  certain  type  than  can  be  provided  in  a  certain  time  frame.  This  situation 
requires  a  decision  that  basically  answers  "who  gets  what,  when."  A  schematic  representation  of 
this  type  of  decision-making  problem  is  provided  in  Figure  13. 

The  diagram  shows  Resource  Users  that  are  combat  units  (such  as  divisions,  wings, 
squadrons,  etc.),  a  System  that  can  be  elaborate  (such  as  a  single  mode  or  multiple  nodes). 
Resource  Pools  (such  as  depots,  air  logistics  centers,  wings,  etc.),  and  a  Transport  System  (that 
can  be  more  or  less  involved  and  includes  air,  sea,  and  land  transport  units).  Decisions  are 
triggered  by  request  messages  coming  from  resource  users  into  the  system  where  the  decision 
maker  sits.  The  decisions  made  by  the  latter  are  implemented  by  means  of  messages  that  task 
resource  pools  and  the  transport  system  to  provide  resources.  These  messages  then  trigger  a  flow 
of  resources  to  the  requesting  user.  The  entire  system  is  embedded  in  an  environment  that  is 
subject  to  a  number  of  uncontrollable  factors  such  as  the  actions  of  the  opposing  forces,  politics, 
weather,  geography,  and  so  on. 


58 


Figure  13.  Schematic  Representation  of  the  Decision-Making  Domain  for  the  Targeted  Class  of 
Decision-Making  Tasks 


Guidance,  Politics  Weather 


This  type  of  decision  problem  occurs  in  many  manifestations  in  all  services  and  on  many 
levels.  For  example,  it  occurs  at  air  support  operation  centers,  where  Army  units  request  air 
resources.  It  is,  without  a  doubt,  a  universal  type  of  tactical  decision-making  problem.  It  is  also 
a  decision  task  that  definitely  has  a  very  high  frequency  of  performance,  a  high  number  of 
performers,  can  quickly  become  very  difficult  to  perform  and  difficult  to  learn,  involves  high 
stakes,  and  is  (in  most  cases)  very  poorly  trained,  if  at  all.  This  type  of  decision-making  task  is 
therefore  clearly  the  right  kind  of  target  for  this  project  and  is,  in  fact,  the  task  type  we  will  be 
focusing  on. 
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The  Terminal  Training  Objective  for  the  project  can  be  formulated  as  follows: 


TERMINAL  TRAINING  OBJECTIVE 


Action 


Conditions 


The  student  will  make  and  implement  decisions 


given  demand-and-supply  problems  presented  within  the  context  of 
a  computer-based  conflict  simulation,  where  the  problems  have  the 
same  time  line  and  uncertainty  profiles  as  the  real-world  target  tasks 


Standards 


and  given  eliciting  stimuli,  information  resources,  and  response 
implementation  facilities  that  are  functionally  equivalent  to  those  of 
real-world  tasks 


with  the  student's  decision  being: 
reliably  effective, 

logically  consistent  with  available  information,  and 
within  the  available  time  window. 


4.3.  Training  Content 


The  next  area  of  concern  is  the  content  of  instruction.  There  are  basically  two  major 
issues  that  must  be  decided  by  the  instructional  designer.  The  first  is  the  meta-content  issue 
which  deals  with  whether  instruction  should  include  content  about  decision  making  in  general. 
The  second  is  the  domain  content  issue,  which  deals  with  whether  content  should  be  tailored  to 
each  specific  domain  in  which  the  targeted  task  type  occms  and  with  what  should  be  included  or 
excluded  from  domain  content. 
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4.3.1.  Instructional  Design  Guidelines 

A  number  of  the  instructional  design  guidelines  developed  earlier  bear  on  the  content 
issue.  These  guidelines  are  reproduced  in  Table  9. 


Table  9.  Instructional  Design  Guidelines  Relevant  to  the  Training  Content  Issue 


Instructional  Design  Guidelines  for  Training  Content 

Guideline 

lie, 

1 .  Include  in  the  training  system  the  capability  to  train  students  in  the  use  of  the  uncertainty  and  time  line  models. 

5.  Train  students  how  to  avoid  and/or  overcome  deficiencies  by  using  practice  situations  in  which  deficiencies 
are  likely  to  show  up. 

6.  Train  students  to  cope  with  limitations  by  including  explanations,  procedural  aids,  and  appropriate  practice 
and  feedback  in  situations  where  limitations  are  likely  to  manifest  themselves. 

9.  Include  in  the  training  system  the  capability  to  train  students  in  using  the  process  model  for  decision  making. 

10.  To  facilitate  Transition  No.  1,  establish  a  foundation  of  domain-specific  knowledge  and  procedural  skills  using 
some  form  of  CAI.  Transition  No.  1  is  complete  when  the  learner  can  reliably  solve  simple  decision-making 
problems  without  aid. 

16.  Training  content  should  include  relevant  technical  domain  knowledge  and  meta  knowledge  on  decision 
making.  Where  appropriate,  the  content  of  either  type  should  be  cast  in  the  form  of  mental  models  of 
prescribed  performance.  Where  appropriate,  such  content  should  also  include  contrasting  good  and  bad 
examples.  Training  content  may  include  study  skills. 


4.3.2.  Application 

Four  types  of  meta-content  can  potentially  be  included  in  training.  These  are  presented 

below. 

1.  Decision-making  models  refers  to  the  kind  of  knowledge  Aagard  and  Braby  (1976) 
call  "decision-making  strategy."  It  is  the  kind  of  knowledge  represented  by  the 
uncertainty,  time  line,  and  process  models,  and  it  includes  the  notions  of  novice  (ROC 
process)  versus  expert  (RB  process)  decision-making. 

2.  Human  Performer  Models  refers  to  knowledge  regarding  human  strengths  and 
weaknesses  (deficiencies  and  limitations)  in  decision  making.  It  also  refers  to 
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knowledge  of  methods  for  avoiding  human  foibles  and  for  capitalizing  on  human 
strengths. 

3.  Management  Skills  refers  to  methods  for  managing  one's  own  cognitive  and 
affective  resources  for  optimal  performance  under  stress. 

4.  Study  Skills  refers  to  methods  for  using  the  training  system  to  one's  own  best 
advantage. 

The  issue  that  needs  to  be  decided  is  whether  one  or  more  of  these  types  of  meta-content 
should  be  included  in  instruction.  On  principle,  one  would,  of  course,  opt  for  inclusion  if  such 
content  increases  instructional  effectiveness  and/or  efficiency.  Whether  knowledge  of  meta¬ 
content  makes  a  difference  in  the  rate  of  skill  acquisition  and/or  in  long-term  parformance  is 
ultimately  an  empirical  question.  There  are  very  good  logical  and  empirical  reasons  to  believe 
that  meta-content  could  be  helpful,  particularly  the  decision-making  models  presented  in  this 
paper.  Meta-content  helps  direct  the  student's  and  the  performer's  attention  and  effort  and  has 
been  found  beneficial  in  a  variety  of  studies  on  problem  solving  and  troubleshooting.  Since  the 
available  evidence  is  not  conclusive  and  the  training  system  is  intended  as  a  research  test-bed, 
there  is  no  need  to  decide  this  question  on  an  a  priori  basis.  The  training  system  will  simply  have 
to  be  designed  such  that  various  t5q)es  of  instructional  treatments  can  be  "dialed  up,"  and  this 
capability  will  simply  have  to  include  the  individual  addition  or  deletion  of  each  of  the  four  types 
of  meta-content. 

But  what  about  domain  content?  Domain  content  is  the  technical  knowledge  of  the 
decision-making  domain  (i.e.,  the  facts,  concepts,  procedures,  rules,  principles,  mental  models, 
and  so  on,  that  describe  the  objects  in  the  decision-making  domain  and  their  interrelationships). 
Clearly,  this  type  of  knowledge  is  a  prerequisite  for  instruction  for  decision-making  skills;  that  is, 
students  must  either  have  it  or  learn  it.  Unfortunately,  this  type  of  content  is  beset  with 
significant  problems:  it  is  full  of  local  differences  (i.e.,  it  is  very  different  for  each  agency 
and  for  each  position)  and  it  is  very  unstable  (i.e.,  it  changes  very  frequently).  Accommodating  a 
wide  range  of  content  variations  and  keeping  up  with  all  content  changes  would  be  logistically 
difficult  and  prohibitively  expensive. 

If  "real"  domain  content  (i.e.,  a  real  domain)  cannot  be  used,  what  then?  The  decision 
maker  must  make  decisions  about  something;  that  is,  this  person  must  operate  in  some  domain. 
The  choices  are  to  use  either  a  domain  that  is  very  stable  and  common  or  one  that  is  artificially 
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created.  The  disadvantage  of  a  stable,  common  domain  is  that  the  students  would  already 
possess  the  prerequisite  domain  knowledge.  The  effect  of  variations  in  prerequisite  training 
could  therefore  not  be  investigated  with  such  a  system.  That  leaves  the  choice  of  an  artificial 
domain  which  is  functionally  and  physically  similar  to  a  number  of  real  domains. 

A  decision  to  go  with  an  artificial  domain  has  transfer  of  training  implications  that  must 
be  clearly  spelled  out.  The  argument  is  that  people  can  be  trained  to  be  better  decision  makers  in 
the  real  world  through  training  in  an  artificial  world,  as  long  as  the  practice  tasks  and  the  real- 
world  tasks  have  the  same  critical  features.  The  uncertainty  and  time  line  models  provide  the 
tools  to  ensme  training  tasks  and  job  tasks  are  indeed  rigorously  similar  in  all  essential  aspects. 
As  long  as  this  similarity  can  be  maintained,  there  is  no  reason  to  expect  anything  other  than 
positive  transfer  of  training  from  the  artificial  world  to  the  real  world.  The  question  is  not  the 
direction  but  the  degree  of  transfer  and  thus  the  cost  vs.  benefit  ratio.  Of  course,  it  may  be 
argued  that  any  benefit  is  worth  the  cost  because  the  cost  of  the  alternative  (i.e.,  the  cost  of 
developing  and  maintaining  multiple  high-fidelity  simulations)  is  prohibitive. 

Besides  the  risk  of  low  transfer,  there  is  another  downside.  If  the  training  system 
incorporates  an  artificial  domain,  it  will  do  nothing  to  alleviate  the  training  problem  units 
have  with  respect  to  "local"  domain  knowledge.  The  training  system  was,  at  least  initially,  seen 
as  a  means  to  provide  the  entire  spectrum  of  training  from  simple  factual  domain  knowledge  to 
the  pinnacle  of  decision-making  skills  under  high-stress,  war-like  conditions.  The  decision  to  go 
to  an  artificial  training  world  enables  the  system  to  provide  the  upper  end  of  the  training 
spectrum,  but  it  eliminates  its  capabilities  at  the  lower  end.  This  is  essentially  the  price  for  a 
system  that  is  instantaneously  applicable  to  any  real  domain  where  the  targeted  task  type  occurs, 
as  opposed  to  a  system  that  can  do  "cradle-to-grave"  training  but  only  in  a  very  narrow, 
specialized  domain  and  only  if  it  is  constantly  updated. 

Whether  the  domain  represented  in  the  training  system  will  be  artificial  or  not,  the  learner 
must  acquire  a  functionally  accurate  mental  model  of  the  domain  within  which  they  will  have  to 
make  decisions.  What  is  meant  by  such  a  model  is  shown  in  Figure  13,  where  the  general 
structure  of  the  "system"  is  depicted,  within  which  the  target  class  of  decision  tasks  must  be 
performed.  This  model  represents  the  highest  level  of  content  or  domain  knowledge  that  the 
learner  must  have  readily  available.  All  other  content  can  essentially  be  deduced  from  this 
model.  For  example,  the  learner  has  to  have  more  detailed  models  for  all  the  subsystems  (boxes) 
in  the  diagram.  The  learner  must  know  the  features,  capabilities,  and  limitations  of  object  classes 
and  individual  objects  in  each  subsystem.  They  must  know  how  objects  within  the  subsystems 
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and  across  subsystems  interact.  They  must  know  everything  that  will  usually  affect  a  decision 
within  the  domain  and  quite  a  few  things  that  will  rarely  or  never  have  any  bearing  on  a  decision. 

They  must  also  be  able  to  tell  the  necessary  from  the  unnecessary  knowledge  because  that  is  one 
problem  encountered  in  the  real  world. 

The  entire  content  issue  can  be  summed  up  as  follows. 

t 

1 .  The  content  of  instruction  must  at  least  include  domain  content  and  may  include 

meta-content.  r 

2.  The  technical  decision-making  domain  will  be  an  artificial  domain  with  the  same 
structural  and  functional  characteristics  as  the  real  domains  in  which  the  target  type  of 
decision-making  task,  the  "demand  and  supply"  task,  occurs. 

3 .  Domain  content  will  consist  of  "mental  models"  of  the  decision-making  domain  and 
of  the  constituent  facts,  concepts,  procedures,  and  principles.  - 

4.  Meta-content  consists  of  four  types.  The  training  system  will  allow  the  inclusion  of 
one  or  more  types  of  meta-content,  depending  on  research  objectives. 

4.4.  Training  Strategy 

This  section  addresses  the  organizational  strategy  for  training  decision-making  skills.  We 
begin  by  recalling  the  instructional  design  guidelines  developed  earlier,  then  progress  to 
strawman  macro-and  micro-organizational  strategies.  The  purpose  of  this  report  is  to  provide 
generic  rather  than  specific  strategies.  The  intent  is  to  describe  the  principle  rather  than  to  show 
the  details  of  its  application  to  a  specific,  invented,  artificial  domain.  Application  details  will  be 
the  topic  of  future  technical  reports. 

4.4.1.  Instructional  Design  Guidelines 

Most  of  the  instructional  design  guidelines  developed  in  the  section  on  theoretical 
foundations  apply  to  the  issue  of  organizational  strategy.  For  the  sake  of  convenience,  these 
guidelines  are  reproduced  in  Table  10. 
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Table  10.  Instructional  Design  Guidelines  for  Organizational  Strategy 


Instructional  Design  Guidelines  for  Organizational  Strategy 

Guideline 

No. 

2.  Ensure  that  practice  problems  have  the  same  uncertainty  and  time  line  profiles  as  the  target  job  tasks. 

3.  Ensure  that  training  provides  for  a  practice  environment  that  features  the  same  types  of  information  sources 
and  requires  the  same  kinds  of  information  access  procedures  as  the  target  environment. 

4.  During  practice,  gradually  increase  time  constraint,  complexity,  and  uncertainty  to  levels  encountered  in  the 
target  job  environment. 

5.  Train  students  how  to  avoid  and/or  overcome  deficiencies  by  using  practice  situations  in  which  deficiencies 
are  likely  to  show  up. 

6.  Train  students  to  cope  with  limitations  by  including  explanations,  procedural  aids,  and  appropriate  practice 
and  feedback  in  situations  where  limitations  are  likely  to  manifest  themselves. 

7.  Provide  high  levels  of  increasingly  complex  practice  in  realistic  decision-making  situations. 

8.  Adapt  instructional  treatments  to  learner  proficiency  by  gradually  compiling  and  finally  withdrawing 
instructional  cues. 

10.  To  facilitate  Transition  No.  1,  establish  a  foundation  of  domain-specific  knowledge  and  procedural  skills  using 
some  fonn  of  CAL  Transition  No.  1  is  complete  when  the  learner  can  reliably  solve  simple  decision-making 
problems  without  aid. 

1 1 .  To  facilitate  Transition  No.  2,  develop  integrated,  competent  levels  of  decision-making  performance  using 
some  form  of  realistic,  dynamic  simulation.  Transition  No.  2  is  complete  when  the  learner  can  reliably  solve 
complex  decision-making  problems  without  aid. 

12.  Do  not  attempt  to  transition  a  learner  from  the  COMPETENT  stage  to  the  EXPERT  stage  (Transition  No.  3) 
by  means  of  formal  training.  Leave  the  development  of  expertise  to  informal  OJT. 

13.  CAI  for  Transition  No.  1  must  include  practice  with  step-by-step  guidance  through  simple  decision  problems 
and  with  a  priori,  artificial  feedback. 

14.  Simulation  for  Transition  No.  2  must  include  realistic,  unaided  practice  with  both  artificial,  a  priori  feedback 
and  natural,  a  posteriori  feedback.  As  the  learner  gains  proficiency,  artificial  feedback  should  be  withdrawn. 

15.  Design  the  organizational  strategy  for  training  decision-making  skills  using  Reigeluth’s  elaboration  strategy  as 
guidance  for  the  macro  strategy  and  Merrill's  CDT  for  the  micro  strategy.  Complement  either  with  elements 
based  on  Keller's  theory  on  motivational  design  of  instruction. 

17.  Practice  should  be  realistic  and  frequent,  cover  a  wide  range  of  difficulty  and  significant  cues,  and  provide 
constructive  feedback  that  is  directly  related  to  training  objectives. 

18.  Enhance  student  motivation  by  getting  the  student's  attention,  establishing  relevance,  setting  attainable  goals, 
providing  for  ample  practice  and  feedback,  and  permitting  learner  control  whenever  possible. 

19.  Enhance  the  presentation  of  instructional  content  with  orientation  aids,  summaries  and  syntheses,  and  with 
clear,  appropriate  illustrations  of  various  types. 

20.  During  early  stages  of  learning,  student  anxiety  should  be  kept  low.  Later  stages  of  learning  should  begin  to 
introduce  realistic  levels  of  stress. 
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4.4.2.  Application:  Macro  Strategy 


For  the  macro  aspects  of  organizational  strategy,  we  rely  on  Reigeluth's  Elaboration 
Theory  (ET)  as  the  primary  organizational  principle.  This  theory  integrates  and  synthesizes 
instructional  sequences  proposed  by  a  number  of  other  researchers  (e.g.,  Ausubel,  Bruner, 

Scandura,  Merrill,  and  Norman  )  "into  an  internally  consistent  set  of  prescriptions  that  were  all  * 

guided  by  the  goal  of  building  stable  cognitive  structures  in  a  meaningful,  subsumptive 

(Ausubel),  or  assimilative  (Mayer)  way"  (Reigeluth,  1987,  p.  245).  It  provides  an  admirable  ^ 

level  of  operational  guidance  for  instructional  designers  in  "selecting,  sequencing,  synthesizing, 

and  summarizing  instructional  content"  (Reigeluth,  1987,  p.  245).  It  explicitly  addresses 

interrelationships  within  instructional  content,  and  it  is  the  only  theory  that  "specifically  allows 

for  some  learner  control  over  the  selection  and  sequencing  of  the  content"  (Reigeluth,  1987,  p. 

246). 


Reigeluth's  ET  prescribes  seven  major  strategy  components  (p.247): 

1 .  An  elaborative  sequence  for  the  main  structure  of  a  course  (and  curriculum) 

2.  A  variety  of  prescriptions  for  sequencing  within  individual  lessons  of  a  course 
(including  learning  prerequisite  sequence) 

3.  Summarizers 

4.  Synthesizers 

5.  Analogies 

6.  Cognitive  strategy  activators 

7.  A  learner  control  format 

The  key  prescription  is  the  elaborative  sequence,  which  is  a  simple-to-complex  sequence 
that  begins  instruction  with: 

a  special  kind  of  overview  containing  the  simplest  and  most  fundamental  ideas,  called  the 
"epitome"  (because  it  epitomizes  the  content).  Then,  subsequent  lessons  add  complexity 
or  detail  to  a  part  or  aspect  of  the  overview  in  layers  (called  "elaborations"). 

Reigeluth  is  fond  of  using  a  zoom  lens  analogy  to  explain  the  concept:  one  starts  with  a 
simplified  overview  representing  the  "big  picture"  (perhaps  something  like  the  picture  presented 
in  Figure  13)  that  is  the  epitome.  Subsequent  lessons  then  zoom  in  on  the  parts  of  that  picture 
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and  present  increasingly  more  elaborate  pictures  of  the  parts.  This  "zooming"  or  elaborating 
continues  in  layers  or  levels  until  the  required  level  of  detail  and/or  complexity  has  been  reached. 

The  elaborative  strategy  makes  intuitive  sense,  has  considerable  theoretical  and  empirical 
support  (Reigeluth,  1983),  and  fits  the  instructional  problem  we  are  attempting  to  solve.  That 
learning  should  proceed  firom  the  simple  to  the  complex  is  simply  a  matter  of  common  sense. 

*  Our  limited  information  processing  capabilities  demand  that  complexity  be  built  up  gradually;  at 

the  beginning,  the  learner  will  only  understand  and  retain  simple  things.  That  a  simplified 
overview,  an  epitome,  should  be  that  beginning  is  logically  convincing  and  supported  by 
cognitive  learning  theories  which  argue  that  the  epitome  provides  a  scaffold  for  the  integration  of 
the  elaborations  to  follow.  This  integration  is  further  supported  by  the  use  of  the  synthesizer  and 
summarizer  elements.  Elaborative  sequencing  starting  with  an  epitome  can  be  applied  very 
neatly  to  the  training  of  decision-making  skills:  the  learner  begins  by  making  the  least  complex 
decisions  possible  and  progresses  through  levels  of  elaboration  to  increasingly  complex  ones. 

The  decisions  at  the  beginning  should  at  once  be  simple  and  represent  the  essence  of  the  type  of 
decision-making  tasks  that  will  be  encountered  in  the  job  being  trained  for;  that  is,  they  should 
"epitomize"  the  task  to  be  learned. 

Reigeluth  has  refined  the  concept  of  an  elaborative  sequence  into  specific  sets  of 
prescriptions  that  apply  to  three  different  types  of  content:  conceptual,  procedural,  and 
theoretical  contents  Consequently,  he  distinguishes  between  conceptual,  procedural,  and 
theoretical  elaborations.  He  further  proposes  that  the  entire  content  of  instruction  towards  a 
particular  goal  should  first  be  organized  on  the  basis  of  a  single  type  of  elaboration  that  fits  the 
primary  content  of  the  instruction,  which  is  then  called  the  organizing  content.  All  other 
content  is  called  supporting  content  and  is  "plugged  into"  the  skeleton  of  the  organizing  content 
wherever  it  is  most  relevant. 

The  organizing  content  for  training  decision-making  skills  is  probably  best  characterized 
as  procedural  content  that  should  be  organized  along  the  lines  of  a  procedural  elaboration.  The 
following  paragraph,  quoted  from  Reigeluth  (1987),  explains  the  notion  of  procedural 
elaboration: 

Procedural  Elaboration.  On  the  other  hand,  the  goals  of  your  comse  might  be  primarily 
procedural(addressing  the  "how"),  as  in  an  English  composition  course.  In  this  case  the 
elaboration  sequence  should  follow  the  optimal  process  of  procedural  skill  acquisition. 
Your  first  activity  as  a  designer  is  to  identify  the  simplest  possible  version  of  the  task 
(usually  equivalent  to  the  shortest  "path"  through  the  procedure  in  P.F.  Merrill's  1980 
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path  analysis  methodology)  and  to  identify  the  "simplifying  assumptions"  that  define  that 
simplest  version.  Your  next  task  is  to  design  the  instructional  sequence  by  gradually 
relaxing  the  simplifying  assumptions  in  the  order  of  most  important,  comprehensive,  and 
fundamental  ones  first,  such  that  progressively  more  complex  paths  are  taught.  Then,  the 
other  types  of  content,  including  concepts,  principles,  learning  prerequisites  and 
remember-level  information,  are  "plugged  into"  that  sequence  at  the  point  where  each  is 
most  relevant,  (p.249) 

The  entire  content  of  a  course  of  instruction  in  decision  making  can  thus  be  organized  in 
levels  where  students  on  the  first  or  epitome  level  to  make  the  simplest  types  of  decisions  that  are 
possible  in  a  given  domain  and  where  they  learn  to  make  more  and  more  complex  decisions  on 
subsequent  levels  of  elaboration. 

The  notion  of  levels  of  elaboration  is  compatible  with  the  multistage  learning  model 
presented  in  Section  3.3.  The  macro  aspects  of  organizational  strategy  can  be  organized  so  that 
learners  progress  on  each  level  from  the  NOVICE  stage  to  the  COMPETENT  stage.  Learners 
thus  first  become  competent  decision  makers  in  the  simplest  and  most  "benign"  version  of  a 
particular  domain,  then  progress  to  the  next  level,  where  they  are  exposed  to  a  more  complex  and 
less  benign  version  of  the  same  domain,  and  so  forth.  On  each  next  higher  level,  learners  are 
again  NOVICEs  with  respect  to  new  facts,  concepts,  procedures,  and  principles  that  they  must 
first  acquire  then  apply  to  the  more  complex,  more  uncertain,  and  more  time-constrained 
decision-making  situations  on  that  level  xmtil,  eventually,  they  are  COMPETENT  on  that  level. 

Each  level  must  therefore  provide  appropriate  instruction  to  facilitate  Transition  No.  1 
(NOVICE  to  ADVANCED),  then  Transition  No.  2  (ADVANCED  to  COMPETENT). 

Guidelines  No.  9  and  10  prescribe  some  form  of  CAI  for  Transition  No.  1  and  some  form  of 
realistic  dynamic  simulation  for  Transition  No.  2. 

Consequently  the  strawman  macro  organizational  strategy  for  a  course  in  logistics 
decision  making  is  based  on  two  organizing  principles.  The  first  is  the  principle  of  levels  of 
elaboration  where  complexity,  uncertainty,  and  time  constraints  rise  with  each  level.  The  second 
organizing  principle  is  a  division  of  each  level  into  two  successive  sections,  where  the  first 
section  is  designed  to  accomplish  Transition  No.  1  in  a  CAI  environment  and  the  second  section 
is  designed  to  accomplish  Transition  No.  2  in  a  simulation  environment.  This  macro  strategy 
concept  is  presented  graphically  in  Figure  14. 


68 


Figure  14.  Macro  Organizational  Strategy  for  a  Course  in  Decision-Making  Skills  (Strawman) 
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The  next  step  prescribed  by  Reigeluth's  ET  is  the  sequencing  of  content  within  lessons 
and  the  integration  into  the  lesson  sequence  of  other  components,  such  as  summarizers, 
synthesizers,  analogies,  cognitive  strategy  activators,  macro-level  motivational  components,  and 
macro-level  learner  control  options.  Reigeluth's  prescriptions  for  within-lesson  sequencing  are 
simple  and  logical.  We  intend  to  follow  them  (without  repeating  them  here)  by  dividing  each 
lesson  into  one  or  more  segments,  where  each  segment  either  provides  instruction  on  a  single 
idea  or  objective  or  represents  a  summarizer,  s)mthesizer,  cognitive  strategy  activator,  analogy, 
or  motivational  component.  Synthesizers,  summarizers,  and  cognitive  strategy  activators  will 
also  appear  as  lessons  (which  will  consist  of  a  single  segment). 

4,4.3.  Application:  Micro  Strategy 

The  next  concern  in  defining  organizational  strategy  is  the  micro  strategy.  Micro  strategy 
concerns  the  organization  of  instruction  within  lessons  and  lesson  segments  and  the  organization 
of  instruction  within  exercises.  For  the  former,  we  rely  primarily  on  Merrill's  Component 
Display  Theory  (CDT)  and  on  Gagne's  nine  events  of  instruction.  For  the  latter  we  rely  on  the 
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dictates  of  functional  realism  and  on  the  guidance  gleaned  from  earlier  analyses.  We  will  first 
address  micro  strategy  within  lessons  and  lesson  segments,  then  turn  to  a  more  detailed  account 
of  strategy  within  exercises. 

Micro  Strategy  within  Lessons  and  Lesson  Segments.  Instructional  theorists  appear  to 
agree  more  on  the  elements  from  which  to  compose  a  micro  strategy  than  on  the  elements  that 
should  be  included  in  a  macro  strategy.  Gagne  and  Merrill  have  each  defined  a  rather  » 

comprehensive  set  of  elements,  either  one  of  which  has  proved  sufficient  for  the  specification  of 
micro  strategies  and  for  the  design  of  effective  and  efficient  instruction.  Keller  focuses  on  the 
motivational  aspects  of  micro  strategy  and  contributes  several  elements  that  are  unique. 

Our  effort  in  defining  micro  strategy  is  guided  by  the  dual  purposes  that  the  training 
system  must  satisfy  (i.e.,  training  as  well  as  research).  We  are  therefore  interested  in  defining  a 
comprehensive  set  of  micro  structure  elements  to  enable  the  investigation  of  a  broad  spectrum  of 
organizational  strategy  issues  at  the  micro  level.  In  other  words,  the  objective  here  is  not  to 
define  the  definitive  micro  strategy;  instead,  the  goal  is  to  provide  researchers  with  a  "kit"  from 
which  they  can  construct  a  wide  range  of  desired  micro  strategies. 

In  Merrill's  CDT,  micro  strategy  (strategy  for  a  single  objective)  is  first  subdivided  into 
the  three  successive  "phases"  of  presentation,  practice,  and  performance  (i.e.,  testing).  This 
scheme  leaves  out  instructional  elements  that  should  occur  prior  to  presentation,  which  other 
theorists  (notably  Gagne,  Keller,  and  Ausubel)  consider  important.  Therefore,  it  appears  that  an 
alternative  "phase"  scheme,  which  includes  such  elements,  would  be  more  comprehensive.  The 
phase  division  suggested  here  consists  of  a  pre-instructional  phase,  instructional  phase,  and 
post-instructional  phase.  The  instructional  phase  includes  Merrill's  presentation  and  practice 
phases,  and  the  post-instructional  phase  includes  Merrill's  performance  phase.  The  types  of 
instructional  micro  elements  that  are  either  mandatory  or  optional  components  of  each  phase  are 
shown  in  Table  11. 
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Table  11.  Micro  Strategy  Elements 


Phases  and  Elements 

Code 

Option 

Function 

Theory 

Support 

PRE-INSTRUCTIONAL 

PHASE 

Prepare  student  for  instruction 

Attention  Grabber 

Ag 

yes 

Get  the  student's  attention 

Keller,  Gagn^ 

Advance  Organizer 

Ao 

yes 

Provide  cognitive  scaffold 

Ausubel 

Recall  Stimulator 

Rs 

yes 

Activate  memory  of  prerequisite 
knowledge  and  skills 

Gagn6,  Keller 

Objective 

0 

no 

Direct  attention  to  desired  outcome 

Gagn6,  Keller 

INSTRUCTIONAL 

PHASE 

Instruct  student 

Generality 

G 

no 

Provide  the  information  required  to 
learn  and/or  control  the  desired 
performance 

Merrill,  Gagne 

Help 

H 

yes 

Facilitate  acquisition  and/or  retention  . 
of  generality 

Merrill,  Gagn^ 

Example 

eg 

yes 

Facilitate  acquisition  and/or  retention 
of  generality 

Merrill,  Gagne 

Demonstration 

D 

yes 

Facilitate  acquisition  and/or  retention 
of  generality 

Merrill,  Gagn6 

Practice  and  Feedback 

P 

no 

Elicit  the  performance  and  provide 
feedback 

Merrill,  Gagn6 

POST-INSTRUCTIONAL 

PHASE 

Solidify  results  of  instruction  and 
evaluate  success 

Summarizer 

Su 

yes 

Facilitate  retention  of  generality 

Reigeluth 

Synthesizer 

Sy 

yes 

Facilitate  connections  to  prior  learning 

Reigeluth 

Test 

T 

no 

Evaluate  the  results  of  instruction 

Merrill,  Gagn6 

Whether  a  particular  element  should  be  included  in  a  micro  strategy  and  the  specific 
content  and  form  that  each  micro  element  should  take  is,  in  all  instructional  design  theories,  a 
fimction  of  the  instructional  objective  that  is  to  be  achieved.  We  advocate  a  strawman  that  relies 
on  the  guidance  provided  by  Merrill  and  Keller;  at  the  same  time,  it  should  be  made  clear  that  the 
ultimate  choice  of  strategy  and  the  authoring  of  the  individual  elements  must  remain  the 
prerogative  of  the  researchers  who  will  use  the  system. 


The  micro-level  organizational  strategy  within  lessons  and  within  lesson  segments  is 
illustrated  in  Figure  15.  The  figure  shows  a  three-phase  organization  on  the  segment  level  and 
within  segments.  The  sequencing  of  segments  within  a  lesson  and  the  sequencing  of  elements 
within  a  segment  is  determined  by  management  strategies.  In  general,  sequencing  will  be  driven 
by  prerequisite  relationships,  degree  of  learner  control,  and  student  achievement. 
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Figure  15.  Micro-Organizational  Strategy  within  Lessons  and  Lesson  Segments 
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Micro  Strategy  within  Exercises.  The  simulation  provides  scenario-based  exercises  that 
afford  students  the  opportunity  to  apply  the  knowledge  and  skill  acquired  via  CAI  to  the  solution 
of  decision  problems.  The  degree  to  which  these  skills  will  transfer  to  actual  job  situations 
depends  first  of  all  on  the  functional  realism  of  these  exercises;  it  does  not  depend  on  how  well 
the  exercises  capture  the  physical  details  of  any  environment  in  which  the  target  "Demand  and 
Supply"  problem  occurs,  but  on  how  well  they  capture  the  underlying,  common  or  general, 
functional  details.  The  degree  to  which  these  exercises  are  instructionally  effective  and  efficient 
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(i.e.,  the  degree  to  which  they  support  the  acquisition  of  decision-making  skills)  will  depend  on 
the  degree  of  instructional  control  that  can  be  overlaid  on  these  exercises. 

The  requirements  for  functional  fidelity  can  be  satisfied  by  providing  a  simulation  based 
on  a  model  that  represents  a  composite  functional  abstract  of  the  real  domains  to  which  the  skills 
acquired  in  our  training  system  are  intended  to  transfer.  Whether  Air  Force  trainers  and  trainees 
♦  will  accept  the  simulation  depends  on  how  obvious  the  relationship  of  the  simulation  is  to  the 

jobs  they  have  to  train  for  (i.e.,  the  simulation  must  be  an  obvious  functional  analogy,  especially 
since  it  cannot  be  an  obvious  physical  analogy).  Thus,  from  the  perspective  of  system  designers 
of  the  system,  the  simulation  must  be  so  designed  that  decision  problems  are  presented  to  the 
decision  maker  by  the  same  kind  of  stimuli  as  in  the  real  world  (i.e.,  by  messages  arriving  over 
various  channels  of  communication).  Furthermore,  the  decision  problems  in  an  exercise  must 
have  the  same  uncertainty  and  time  line  characteristics  as  real-world  decision  problems  and  the 
decision  maker  must  have  access  to  the  same  types  of  information  resources  to  reduce 
uncertainty  as  are  available  in  the  real  world.  It  also  means  the  same  types  of  noise  and 
distractions  must  occur  during  decision  making  as  in  the  real  world,  and  whatever  feedback 
occurs  in  the  real  world  also  occurs  in  kind  and  with  the  same  timing  in  the  simulation. 

The  requirements  for  instructional  control  are  dictated  by  two  factors.  First,  it  will  be 
necessary  to  control  exercise  difficulty  and/or  complexity  in  parallel  with  the  levels  defined  by 
the  macro  strategy.  Second,  it  will  be  necessary  to  control  the  insertion  of  instructional  elements 
designed  to  facilitate  the  acquisition  of  the  various  types  of  meta-skills  for  decision  making. 

Control  of  exercise  difficulty,  or  more  precisely,  control  over  the  difficulty  of  the 
decision  problems  presented  to  students  during  an  exercise,  can  be  achieved  by  manipulating  the 
type  as  well  as  the  amoxmt  of  uncertainty  in  each  problem,  the  timing  of  the  decision  phases  (e.g., 
by  making  the  decision  window  shorter),  the  density  with  which  decision  problems  occur  in  a 
given  time  frame,  and  the  amount  and  types  of  distractions  introduced  into  the  exercises.  These 
four  types  of  control  must  be  available,  whether  or  not  researchers  decide  to  train  meta-content 
(i.e.,  whether  the  instructional  treatment  will  include  explicit  instruction  in  the  time  line  model, 
the  uncertainty  model,  and  the  process  model).  Thus,  researchers  who  will  use  the  system  will 
need  an  interface  that  allows  such  control,  and  the  system  designer  must  build  the  system  to 
satisfy  these  control  and  interface  requirements. 
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Finally,  control  over  the  insertion  of  instructional  elements  associated  with  explicit 
training  of  meta-skills  must  be  available  to  the  researchers;  therefore,  it  must  be  "designed  into" 
the  system  by  the  system  and  instructional  engineers. 

The  preceding  discussion  demonstrates  that  instructional  strategy  within  exercises  (and 
on  the  macro  level  between  exercises)  must  be  a  function  of  considerations  arising  from  the 
combined  needs  for  functional  realism  and  instructional  control.  Both  of  these  needs  are 
considered  in  the  discussion  below,  where  instructional  strategy  is  described  from  the  point  of 
view  that  an  exercise  should  represent  what  decision  makers  do  as  they  "pull  a  shift"  in  their 
assigned  position.  This  is  essentially  the  concept  around  which  SuperKEATS,  the  predecessor 
system,  was  built,  and  (for  lack  of  concrete  information  about  domains  other  than  the  Air 
Support  Operations  Center  (ASOC))  this  is  the  currently  operative  concept  for  the  exercises. 

Exercises  are  divided  into  three  phases:  each  exercise  starts  with  m  Orientation  Phase, 
moves  into  an  Operations  Phase,  and  concludes  with  a  Debriefing  Phase.  The  orientation 
phase  mirrors  the  start  of  the  shift,  the  operations  phase  mirrors  the  remainder  of  the  shift,  and 
the  debriefing  phase  mirrors  nothing  that  occurs  in  reality.  (The  debriefing  phase  is  dictated  by 
the  instructional  need  for  artificial  feedback.)  This  tri-partition,  illustrated  in  Figure  16, 
constitutes  in  effect  the  micro  organizational  strategy  for  exercises.  Below  this  level,  more 
detailed  organization  is  essentially  determined  by  the  requirements  for  functional  realism.  In  the 
following  paragraphs,  these  functional  requirements  are  described  for  each  exercise  phase. 

During  the  Orientation  Phase,  battle  managers  commonly  receive  a  "shift  briefing,"  and 
they  bring  themselves  up-to-date  by  studying  the  latest  available  information.  The  orientation 
phase  includes  a  reassessment  of  the  situation  and  modifications  to  the  existing  plans  of  action. 
The  information  resources  available  for  this  phase  are  briefings,  orders,  status  boards,  and 
message  logs. 

The  instructional  strategy  for  this  phase  must  present  students  with  an  array  of 
information  resources,  very  similar  to  those  that  occior  in  reality.  Students  should  be  able  to 
browse  these  sources  over  a  time  span  that  could  be  limited.  Students  should  then  have  access  to 
a  situation  assessment  (SA)  and  a  plan  of  action  (PA),  either  of  which  should  be  modifiable. 
Students  can  (but  must  not)  receive  immediate  feedback  regarding  the  modifications  they  make. 
All  information  resources  as  well  as  the  original  SA  and  PA  are  scripted  during  exercise 
development.  The  complexity  for  these  items  must  be  controlled  either  by  scripting  guidelines  or 
by  designing  different  templates  for  different  levels. 
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Figure  16.  Micro-Organizational  Strategy  within  Exercises 


The  Operations  Phase  is  entered  after  the  time  limit  for  the  Orientation  Phase  expires  or 
after  students  have  finished  orienting  themselves.  During  this  phase,  the  logisticians  receive 
messages  to  which  they  must  react.  Some  of  these  messages  are  demands  for  resources  from 
operational  units  (i.e.,  from  resource  users),  others  are  replies  to  messages  sent  out,  and  still 
others  inform  the  battle  manager  of  changes  in  the  status  of  either  resource  users  or  resource 
pools  (see  Figure  13)  or  of  changes  in  operational  plans.  Finally,  some  messages  are  totally 
irrelevant. 

The  demand  messages  trigger  decisions  on  whether  to  refuse  or  accept  the  demand,  on  the 
priority  in  which  the  demands  will  be  satisfied,  and  on  the  optimal  methods  to  satisfy  the 
demands.  To  make  these  decisions,  logisticians  must  reduce  secondary  uncertainty  within  the 
available  decision  window  by  consulting  available  information  resources.  Typically,  these 
resources  include  status  boards,  other  members  of  the  battle  staff,  reference  materials,  and 
agencies  outside  the  battle  staff.  Once  the  decision  is  made,  it  is  implemented  by  sending 
messages  that  usually  go  to  resource  pools  as  well  as  to  the  resource  users  who  issued  the 
demand.  Reply  messages  typically  provide  information  that  must  be  "worked  into"  an  ongoing 
decision  process.  Status  and  plan  change  messages  either  trigger  updates  to  status  boards  or 
modification  of  the  SA  and  PA.  Irrelevant  messages  are  ignored.  The  environment  in  which 
these  activities  occur  can  vary  from  completely  quiet  to  extremely  distracting  and  noisy.  The 
density  with  which  messages  arrive  can  be  anything  from  "once  in  a  while"  to  "hot  and  heavy" 
(i.e.,  the  task  load  can  vary  over  a  wide  spectrum). 


75 


The  instructional  strategy  for  this  phase  must,  at  a  minimum,  mirror  these  conditions. 
Messages  of  all  t5T)es  must  be  presented  to  students  in  proportions  and  densities  that  axe 
controllable.  In  SuperKEATS,  density  was  controllable  by  scenario  scripting  (e.g.,  operational 
units  that  had  superior  strength  compared  to  opposing  forces  requested  air  support  less 
frequently)  and  by  changing  the  exercise  clock  (real  time,  twice  real  time,  or  lOx  real  time). 
Information  resources  of  various  types  must  be  available  for  consultation  and,  finally,  message 
forms  for  sending  out  messages  must  be  available.  Noise  and  distracting  information  must  be 
presented  in  a  controllable  manner. 

These  strategy  elements  are  dictated  by  the  requirement  for  fimctional  fidelity.  They 
must  be  complemented  by  elements  arising  from  the  need  for  instructional  control  over  the 
presentation  of  both  domain  and  meta-content.  What  .students  do  in  an  exercise-is  basically 
practice.  All  the  stimiili  and  resources  required  for  appropriate  practice  in  domain  skills  are 
already  supplied  by  satisfying  the  reqmrements  of  functional  fidelity  —  that  is,  all  but  one: 
feedback.  Feedback,  as  indicated  previously,  is  a  critical  issue  because  it  is  an  essential 
instructional  element;  however,  immediate,  a  priori  feedback  (the  most  useful  type)  is  relatively 
difficult  to  provide.  During  the  Operations  Phase,  feedback  should  be  available  immediately 
after  a  decision  is  made,  and  such  feedback  should  provide  evaluation  and  corrective  aid  with 
regard  to  the  solution  of  the  decision  problem  itself  and  with  regard  to  performance  in  terms  of 
the  underlying  model  of  decision  making  (if  meta-content  is  taught  explicitly).  This  type  of 
feedback  must  be  artificially  introduced  and  must  be  controllable  in  terms  of  content  so  as  to 
permit  fading  during  later  stages  of  learning  as  the  student  moves  increasingly  toward  the 
recognition-based  decision  mode.  "Natural"  feedback  occurs  as  a  consequence  of  the  changes 
introduced  into  the  domain  by  the  implementation  of  decisions.  This  type  of  feedback  is 
confounded  with  the  effects  of  other  factors,  usually  delayed  and  difficult  to  attribute  to  a 
specific  decision.  Nevertheless,  it  is  feedback,  it  is  unavoidable,  and  decision  makers  pay 
attention  to  it.  Therefore,  as  the  system  design  progresses,  we  must  be  attentive  to  opportunities 
to  make  such  natural  feedback  more  useful. 

In  addition  to  feedback,  other  types  of  instructional  elements  can  be  introduced  into 
exercises  and  should  be  available  as  means  to  manipulate  the  instructional  strategy.  One  type, 
which  was  used  in  SuperKEATS,  consists  of  suggested  solutions  to  the  decision  problem  at 
hand.  Another  type  are  prompts  of  various  kinds  to  direct  students’  attention  to  salient  features 
of  the  current  decision  problem  to  provide  some  degree  of  assistance.  Such  assistance  can 
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address  domain  content  concerns,  meta-content  concerns,  or  both,  and  it  may  be  introduced 
during  early  stages  of  the  learning  process  and  faded  as  the  learner's  performance  improves. 

A  particularly  interesting  type  of  learning  assistance  may  be  provided  by  using 
immediate,  a  priori  feedback  as  a  device  to  improve  decisions  prior  to  implementation.  In  this 
mode,  students  would  input  a  decision,  then  choose  to  have  it  evaluated  before  committing 
themselves  to  it  (and  before  the  system  works  it  into  the  situation).  If  the  feedback  from  the 
evaluation  indicates  problems  with  the  decision  (either  by  a  low  score  on  some  parameters  or  by 
pointing  to  specific  deficiencies),  students  could  then  modify  their  decision  until  feedback  is 
satisfactory.  This  "exploratory  optimization"  would  provide  the  student  with  a  unique 
opportunity  to  explore  and  try  out  specific  variations  prior  to  making  a  commitment  to  a 
particular  decision.  This  type  of  learning  assistance  not  only  highlights  the  role  of  commitment 
but  also  provides  instant  remediation  for  gaps  and  misconceptions  in  the  learner's  current 
knowledge  base.  If  this  mode  is  used,  it  should  be  used  during  early  stages  and  initially  without 
time  constraints.  As  the  learners  progress,  time  constraints  can  be  introduced  so  that  they  can  do 
all  the  exploring  they  want  up  to  the  default  point.  During  later  stages,  the  mode  should  be 
withdrawn  entirely. 

The  Debriefing  Phase  is  purely  an  instructional  device.  During  this  phase,  learners 
receive  cumulative  feedback  on  the  entire  exercise,  from  its  start  to  its  termination.  This 
feedback  should  address  three  issues:  (1)  set  of  "bean  counting"  scores  that  indicate  how  well 
the  battle  went  for  the  decision  maker's  side;  (2)  statistics  on  how  many  decision  problems  were 
presented,  how  many  decisions  were  actually  made  (and  on  how  many  the  student  "dropped  the 
ball"),  and  the  quality  of  those  decisions;  and  (3)  statistics  that  indicate  how  well  the  student 
performed  in  terms  of  meta-content  (i.e.,  was  uncertainty  reduced  in  the  right  priority?  Were  the 
appropriate  information  resources  accessed?  How  many  decisions  were  made  before  and  after 
the  default  point?  How  long  were  recognition  times?  etc).  Only  the  first  type  of  cumulative 
debriefing  feedback  was  used  in  SuperKEATS. 
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V.  CONCLUSIONS 


The  objective  of  this  report  was  to  lay  the  theoretical  foundation  for  a  research  project 
dedicated  to  the  development  of  an  optimal  training  methodology  for  decision-making  skills  in 
the  arena  of  LC^.  A  training  methodology  is  folly  specified  when  organizational,  delivery,  and 
management  strategy  are  defined  (Reigeluth,  1983).  The  delivery  strategy  for  the  project  was 
prescribed  in  general  terms  as  a  computer-based  training  system  that  can  run  on  common  desktop 
machines  without  the  assistance  of  instructors.  Therefore,  development  of  a  theoretical 
foundation  had  to  address  the  issues  of  organizational  strategy  and  management  strategy  under 
the  constraint  of  a  prescribed  delivery  strategy. 

Organizational  macro  and  micro  strategies  for  a  desktop-computer-based  training  system 
were  defined  by  conducting  analyses  which  examined  four  topics  in  order: 

•  What  is  decision  making? 

•  How  do  people  make  decisions? 

•  How  do  people  learn  to  make  decisions? 

•  What  direct  instructional  design  guidance  is  available? 

The  analyses  produced  a  coherent  and  cohesive  theoretical  framework  based  on  the 
concept  that  decision  making  is,  in  essence,  an  imcertainty  reduction  task  that  requires 
commitment  to  a  course  of  action  in  the  face  of  imavoidable,  residual  uncertainty.  A  series  of 
models  was  developed  that  describes  the  nature  of  the  decision-making  task,  the  process  of 
imcertainty  reduction,  and  the  process  of  learning  decision-making  skills.  Instructional  design 
guidelines  were  deduced  from  the  models  and  firom  other  available  guidance  uncovered  by  the 
analyses.  The  guidelines  were  then  applied  to  the  training  problem  at  hand,  and  a  complete 
general  organizational  strategy  for  the  desktop  training  system  was  developed. 

Instructional  and  system  design  and  development  will  lead  to  a  concrete  training  and 
research  system.  The  system  will  be  loaded  with  a  baseline  training  program.  The  baseline 
training  program  will  feature  an  artificial  logistical  domain  (a  space-based,  science-fiction-like 
context),  and  it  will  be  organized  along  the  organizational  strategy  developed  in  this  report.  It 
will  also  incorporate  a  baseline  management  strategy  consisting  of  a  level  of  learner  control  and 
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advancement  criteria.  Both  strategies,  organizational  and  management,  will  be  modifiable  for 
research  purposes. 

The  primary  purpose  of  empirical  research  will  be  to  optimize  the  training  methodology 
and  to  provide  a  basis  for  generalizability  to  other  decision-making  domains.  Two  significant 
research  issues  have  already  been  mentioned:  the  issue  of  meta-content  and  the  issue  of  transfer 
from  an  artificial  domain.  However,  these  are  by  no  means  the  only  issues  that  will  have  to  be 
investigated.  Table  12  lists  a  number  of  important  research  issues.  This  list  is  provided  with  the 
caveat  that  it  is  not  and  caimot  be  complete  or  exhaustive  at  this  point;  it  is  essentially  a  "starter 
list."  New  issues  will  undoubtedly  arise  during  the  development  of  the  system  and,  especially,  as 
actual  research  gets  underway. 


Table  12.  Starter  List  of  Research  Issues 


No. 

Issue  Title 

Type 

Research  Questions 

1 

Artificial  Domain 

Content 

To  what  extent  does  decision  making  in  a  real 
domain  benefit  from  training  in  an  artificial  domain 
where  the  same  types  of  problems  are  solved? 

2 

Meta-content 

Content 

Does  the  inclusion  of  meta-content  increase  training 
effectiveness  and/or  efficiency? 

What  type  of  meta-content  is  most  helpful? 

3 

Macro  Partitions 

Organization 

What  is  the  optimal  number  of  elaboration  levels  for 
given  entry  and  exit  skills? 

D 

Learner  Control 

Management 

What  are  the  effects  of  various  levels  of  learner 
control  on  training  effectiveness  and  efficiency? 

5 

Presentation 

Form 

Content  Form 

Does  the  form  of  content  presentation  affect  training 
effectiveness? 

6 

Facilitative 

Elements 

Organization 

What  are  the  effects  of  including  facilitative 
elements  on  student  achievement  and  instructional 
efficiency? 

7 

Feedback  Forms 

Organization 

What  are  the  effects  of  a  priori  feedback,  a  posteriori 
feedback,  or  combined  feedback  on  student 
achievement  and  instructional  efficiency? 

8 

Exploration 

Practice 

Organization 

Can  forms  of  exploratory  practice  increase  student 
achievement  and/or  instructional  efficiency? 

9 

Criterion  Levels 

Management 

Where  should  one  set  the  criterion  levels  for  student 
advancement  (to  the  next  segment,  next  lesson, 
simulation  practice,  the  next  level)  for  the  optimal 
compromise  between  effectiveness  and  efficiency? 
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APPENDIX  A.  CHARACTERISTICS  OF  THREE  TYPES  OF  DECISION-MAKING 

TASKS 


Decision  Problem 
Characteristics 

Case  1 

Taxicab  Dispatcher 

Case  2 

Supply  Logistician 

Case  3 

Car  Buyer 

Starting  Situation 

Customer  calls  cab  company 

X.  Needs  cab  to  airport  20 
miles  from  his  house;  must  be 
there  by  6:50.  It  is  now  6:00 
a.m.  Current  traffic  density 
means  the  trip  will  take  30 
minutes  (i.e.,  the  customer 
must  be  picked  up  by  6:20). 

The  dispatcher  (D)  is  busy 
with  20  cabs  en  route,  five 
cabs  sitting  at  airport,  two 
cabs  sitting  elsewhere. 

Supply  logistician  (L) 
receives  request:  squadron  in 
Saudi  Arabia  needs  several 
spare  parts  for  aircraft  within 
five  days.  Severe  operational 
impact  if  they  do  not  get  the 
spares.  No  spare  part 
shortages  stateside.  No 
requests  from  other  units  for 
the  same  types  of  parts. 
Constant  flow  of  air  and  sea 
transports  to  theater  ongoing. 
Plenty  of  available  transport 
capacity.  No  en  route  threats. 

Single  buyer  who  has  an  old 
car  (a  truck).  Car  now  in  the 
shop  more  than  on  the  road. 
Heavy  bills.  Has  decided  to 
buy  a  new  car  but  does  not 
know  what  she  wants, 
although  she  believes  she 
does  not  want  a  truck  again. 
Unfamiliar  with  market.  Can 
buy  up  to  45K  worth  of  car 
without  strain. 

Decision  to  be  Made 

D  must  decide  which  cab  he 
should  dispatch. 

L  must  decide  which  of 
several  possible  supply 

Car  buyer  (CB)  must  decide 
which  car  she  will  buy. 

solutions  he  will  implement. 

Time  Line  Aspects 

Length  of  Decision 
Window 

Seconds,  minutes 

Hours,  days 

Weeks,  months 

Recognition  Process 
(OCP  -  RP) 

OCP  occurs  when  the 
customer  starts  talking  to  D. 

RP  occurs  when  D  recognizes 
the  call  is  a  transportation 
request  he  must  deal  with. 
Recognition  time  is  largely 
dependent  on  how  quickly  the 
customer  gets  to  the  point 
(e.g.:  "Is  this  the  X  cab 
company?  Who  am  I  talking 
to?  My  wife  drove  the  car 
yesterday,  and  I  don't  know 
what’s  wrong  with  it,  but 
today  the  darned  car  won't 
start  and  I  have  to  fly  to 

Podunk  today..."  versus  "I 
need  a  cab  to  the  airport!"). 

OCP  occurs  when  the 
squadron's  request  arrives  at 

L's  office.  RP  occurs  when 
the  request  is  being  read  by  L 
and  L  recognizes  he  needs  to 
deal  with  it  (and  that  he  can 
neither  toss  it  or  hand  it  off). 
Recognition  time  is  highly 
dependent  on  how  many  other 
messages  are  waiting  for  L's 
attention,  on  how  many  other 
things  he  is  attending  to  at  the 
moment,  and  on  how 
organized  he  is. 

OCP  occurs  when  the  cost  of 
maintaining  the  old  car  begins 
to  exceed  the  cost  of  owning  a 
new  car.  RP  occurs  when  CB 
realizes  that  fact.  (OCP  may 
also  occur  when  CB  just  gets 
tired  of  the  old  car  and  wants 
something  new.  In  that  case, 
RP  occurs  when  CB  realizes 
she  cannot  go  on  carrying  her 
desire  around  with  her,  that 
she  has  to  do  something  about 
it.)  In  either  case,  the  OCP 
and  the  RP  are  probably  not 
very  distinct  moments  in  time; 
rather,  they  just  "creep  up"  on 
CB. 
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Decision  Problem 
Characteristics 


Uncertainty 
Reduction 
Process  (RP-DP) 


Implementation 
Process  (ISP-IFP) 


Feedback 
Process  (FSP-FFP) 


Case  1 

Taxicab  Dispatcher 


Uncertainty  reduction  is 
usually  achieved  within  a 
single  working  period.  The 
length  of  that  period  is  most 
likely  less  than  one  minute. 


The  DefP  is  reached  prior  to 
6:20,  when  even  the  closest 
free  cab  carmot  get  to  the 
customer  pickup  on  time. 


There  is  no  implementation 
delay  and  implementation 
itself  takes  as  long  as  a  radio 
call  to  the  selected  cab  (i.e., 
seconds). 


Feedback  (call  from  cab 
departing  customer  location) 
will  come  in  within  minutes 
after  implementation.  Dmust 
stay  on  it  because  the 
dispatched  cab  may  have 
problems  and  a  new  cab  may 
have  to  be  tasked.  Feedback 
ends  with  a  definite  message 
of  customer  delivery  at 
airport. 


Case  2 

Supply  Logistician 


Uncertainty  reduction  is 
usually  achieved  over  several 
working  periods,  lasting 
several  minutes  each.  These 
periods  are  usually  separated 
by  longer  intervening  work 
suspense  periods  during 
which  other  high-intensity 
tasks  are  accomplished.  The 
entire  evolution  from  RP  to 
DP  may  take  a  day  or  two  of 
elapsed  time  and  one  or  more 
hours  of  working  time. 

The  DefP  is  reached  when  the 
time  to  send  parts  exceeds  the 
time  left  to  the  five-day 
deadline. 


There  may  be  implementation 
delays  due  to  "approval" 
requirements  or 
communication  outages. 
Implementation  is 
accomplished  by  messages 
which  take  minutes  to 
compose  and  transmit. 


Feedback  is  not  a  sure  thmg 
in  this  case.  L  may  never 
receive  any  confirmation  that 
anything  is  underway  or  that 
anything  has  gotten  to  the 
destination.  He  can,  however, 
solicit/access  such  feedback 
information.  L  usually  does 
spend  a  good  deal  of  time 
answering  requester  queries 
as  to  just  where  the  shipments 
are  in  the  pipeline. 


Case  3 
Car  Buyer 


There  are  two  basic 
possibilities:  the  rational 
approach  or  the  impulse 
approach.  The  rational 
approach  is  a  long,  protracted 
process  with  numerous  work 
episodes  that  may  last 
anywhere  from  minutes  to 
hours.  The  impulse  approach 
circumvents  the  labor  of 
imcertainty  reduction  in  favor 
of  instant  gratification. 


The  DefP  for  this  case  (with 
either  approach)  is  an 
indistinct,  or  "creeping"  point, 
unless  the  old  car  suddenly 
expires  totally. 


There  can  be  implementation 
delays  (due  to  hesitation) 
lasting  days,  weeks,  or  even 
months.  Implementation  can 
take  a  long  time  (car  . 
availability,  loan  approval). 


Feedback  is  instantaneous  and 
certain  and  usually  quite 
definite.  It  starts  when  CB 
drives  the  car  off  the  dealer’s 
lot  and  continues  until  she 
gets  rid  of  it. 


Uncertainty 
Reduction  Aspects 


Overall  Magnitude  of 
Uncertainty 


Decision  Problem 
Characteristics 

Case  1 

Taxicab  Dispatcher 

Case  2 

Supply  Logistician 

Cases 

Car  Buyer 

Situation  Uncertainty 
(Us) 

Clear  situation  (i.e.,  very  low 
or  zero  Us).  The  dispatcher  is 
“on  top  of  it”  and  knows  how 
many  and  what  kinds  of 
demands  he  has  pending, 
how  many  cabs  he  has 
moving  in  various  areas  of  the 
city  and  how  many  he  has 
sitting,  and  where  and  what 
the  traffic  situation  is  along 
major  routes. 

Clear  situation  (i.e.,  very  low 
or  zero  Us).  The  requirement 
or  need  is  very  clear  and 
specific,  the  supply  and 
transportation  situation  is  well 
known,  and  the  consequences 
of  not  satisfying  the 
requirement  are  known. 

Some  Us.  CB  is  aware  of  the 
high  current  maintenance 
costs  but  does  not  know  for 
sure  whether  owning  a  new 
car  would  be  cheaper. 

Goal  Uncertainty 
(Ug) 

Clear  goals  (i.e.,  very  low  or 
zero  Ug).  Get  the  customer  to 
the  airport  on  time  and  with  a 
minimum  of  unpaid  mileage. 

D  knows  exactly  what  he 
wants  to  achieve. 

Clear  goals  (i.e.,  very  low  or 
zeroUg).  L  knows  exactly 
what  is  to  be  accomplished 
(“Get  the  stuff  to  the  unit 
ASAP,  but  no  later  than  five 
days  from  now!”). 

Very  vague  goals  (i.e.,  very 
highUg).  CB  wants  a  “new 
car”  that  is  not  a  truck  but 
beyond  that  has  not  yet 
determined  whether  she  is 
looking  for  a  particular  type 
of  car,  whether  she  wants  to 
maximize  reliability; 
minimize  price;  maximize 
comfort,  looks,  or 
performance;  etc. 

Option  Set 

Uncertainty  (Uos) 

Relatively  high  Uos.  This  is 
the  primary  area  on  which  D 
needs  to  work.  He  knows  the 
two  basic  option  classes  for 
this  problem:  Option  Class  1 : 
Use  an  “en  route  cab”  that  is 
near  the  customer  location; 
Option  Class  2:  Use  a  sitting 
cab.  He  needs  to  identify 
specific  options  for  at  least 
one  of  the  two  classes. 

Relatively  high  Uos.  L  knows 
basic  option  classes  for  this 
type  of  problem:  Option 

Class  1:  Normal  sourcing  and 
air  transport;  Option  Class  2: 
Lateral  sourcing  in  theater, 
followed  by  resupplying 
lateral  source  if  required.  He 
does  not  have  specific  options 
in  either  class  to  start  with. 

Very  high  Uos.  CB  knows 
there  is  a  bewildering  array  of 
options  out  there,  but  she  can 
not  yet  cut  this  “options 
overload”  down  to 
manageable  size. 

Option  Feasibility 

Uncertainty 

(Uof) 

Some  Uof  D  knows  that 
feasibility  for  Option  Class  1 
is  much  better  than  for  Option 
Class  2;  Cabs  in  the  area  have 
a  much  better  chance  of 
making  the  pickup  deadline. 
But  he  knows  nothing  about 
feasibility  of  specific  options 
because  he  has  none  as  yet. 

Some  Uof  L  knows  that 
Option  Class  1  has  generally 
better  feasibility  than  Option 
Class  2.  He  relegates  Class  2 
to  “fail-back  options”  status. 
Will  have  to  work  on  Uof  for 
each  specific  option. 

Very  low  to  zero  Uof.  CB 
knows  she  can  buy  anything 
up  to  45K  (i.e.,  she  can 
implement  a  very  wide  range 
of  options). 
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Decision  Problem 
Characteristics 


Option  Effects 

Uncertainty 

(Uoe) 


Uncertainty 
Reduction  Strategy 


Case  1 

Taxicab  Dispatcher 


Some  Uoe.  D  knows  the 
effects  of  options  from 
Class  1  are  better  than  the 
effects  of  Class  2  options. 
Cabs  in  the  area  have  a  much 
better  chance  of  getting  to  the 
customer  with  a  minimum  of 
unpaid  miles.  But  he  knows 
nothing  about  effects  of 
specific  options  since  he  has 
none  as  yet. 


D  needs  to  first  identify 
specific  options  for  the 
preferred  class  (i.e.,  he  needs 
to  reduce  Uos  before  anything 
else).  Hence,  he  will  radio  a 
call:  “Is  anyone  in  such-and- 
such  an  area?”  Ifhe  gets  only 
one  response,  he  needs  to  find 
out  whether  that  driver  can 
indeed  take  this  fare  (i.e.,  he 
needs  to  reduce  Uof).  Ifhe 
gets  more  than  one  response, 
he  will  have  to  determine  who 
can  take  the  fare  and  who  is 
closest  (i.e.,  he  needs  to 
reduce  Uof  first,  and  then 
further  discriminate  by 
reducing  Uoe). 


Case  2 

Supply  Logistician 


Some  Uoe.  L  knows  that 
Option  Class  1  has  more 
desirable  long-term  effects 
(more  supplies  in  theater)  and 
that  Option  Class  2  has  good 
short-term  effects  (early 
availability  of  spare  parts). 
Might  have  to  work  on  Uoe 
for  specific  options  if 
feasibility  issues  are  not 
decisive. 


L  needs  to  identify  a  specific 
option  for  the  preferred  class 
and  reduce  Uof  for  that  option 
as  much  as  possible.  Ifhe  can 
achieve  a  high  level  of 
confidence  that  the  option  is 
feasible,  he  can  implement 
that  option  without  putting  a 
fallback  option  from  Class  2 
on  standby.  If  feasibility 
remains  uncertain,  he  should 
implement  with  fallback. 


Case  3 
Car  Buyer 


Very  high  Uoe.  Since  she 
does  not  know  what  she  wants 
to  achieve  and  has  not  yet 
narrowed  her  huge  option  set, 
cannot  determine  what  buying 
a  new  car  will  do  for  her, 
except  reduce  her  current 
maintenance  bills  and 
increase  car  availability. 


CB  needs  to  start  with  Ug  by 
setting  some  clear  goals  (e.g., 
“I  want  a  car  for  getting 
around  town  and  for  little 
weekend  trips.  It  should  be 
economical  first,  comfortable 
second,  and  look  good  third. 

It  should  seat  four  people  and 
a  weekend’s  worth  of  luggage 
for  two.”).  That  will  have  the 
effect  of  reducing  the  huge 
option  set  to  four-door  sedans 
under  45K,  enabling  her  to 
define  a  manageable  set  of 
options.  She  can  then  reduce 
Uoe  for  each  option  based  on 
her  three  criteria  and  end  up 
with  a  rank-ordered  list.  The 
first  car  on  the  list  is  the 
rational  decision. 


APPENDIX  B.  ACRONYMS 


AFHRL 

Air  Force  Resources  Laboratory 

AL/HRTC 

Armstrong  Laboratory,  Human  Resources  Directorate,  Technical 
Training  Research  Division,  Instructional  Design  Branch 

AFLC 

Headquarters  Air  Force  Logistics  Command 

ALC 

Air  Logistics  Center 

ASOC 

Air  Support  Operations  Center 

Command  and  Control 

CAI 

Computer-assisted  Instruction 

CB 

Car  Buyer 

CDT 

Component  Display  Theory 

CINC 

Commander-in  Chief 

D 

Dispatcher 

DefP 

Default  Point 

DP 

Decision  Point 

DS 

Demand  and  Supply 

ECM 

Electronic  Counter  Measures 

ET 

Elaboration  Theory 

FFP 

Feedback  Process 

FSP-FFP 

Feedback  Process 

HQ  USAF/LGX 

Air  Force  Logistics  Plans  and  Program  Directorate 

HR 

Air  Force  Human  Resources  Directorate 

ID 

Instructional  Design 

ISD 

Instructional  Systems  Development 

ISP-IFP 

Implementation  Process 

KR 

Knowledge  of  Results 

L 

Logistician 

LC^ 

Logistics  Command  and  Control 

OCP 

Objective  Choice  Point 

OJT 

On-the-job  Training 

PA 

Plan  of  Action 

PC 

Personal  Computer 

ROC 

Rational  Outcome  Calculation 

RB 

Recognition-Based  Decision  Making 

RMK 

Reigeluth-plus-Merrill-plus-Keller 

RP 

Recognition  Point 

RP-DP 

Uncertainty  Reduction  Process 

RPD 

Recognition-Primed  Decision 

SA 

Situation  Assessment 

SEA 

Systems  Engineering  Associates 

Tc2 

Tactical  Command  and  Control 

UMSA 

Unified  Model  for  Skill  Acquisition 

USAF 

United  States  Air  Force 

Us 

Situation  Uncertainty 

Ug 

Goal  Uncertainty 

Uo 

Option  Uncertainty 

Uos 

Set  Completeness  Uncertainty 

Uos 

Option  Set  Uncertainity 

UoE 

Effects  Uncertainty 

Uoe 

Option  Effects  Uncertainty 

UoF 

Feasibility  Uncertainty 

Uof 

Option  Feasibility  Uncertainity 
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